2024-09-12T07:44:49.417 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-12T07:44:49.433 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T07:44:49.529 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438 branch: wip-dis-testing description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: idryomov@gmail.com first_in_suite: false job_id: '7902438' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: rbd default data pool: datapool rbd default features: 61 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1680 sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 sleep_before_teardown: 0 subset: 42/100 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFVWz4mh4Jf0nGAQ9Jc69Q5d75SVGnKGVSM/5drhDnc1kvuIxWsMRRm9EyLVTEOLSKNDiaibvqbIbHIB2licWtU= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-12_04:46:10 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-12T07:44:49.529 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa; will attempt to use it 2024-09-12T07:44:49.530 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks 2024-09-12T07:44:49.530 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-12T07:44:49.532 INFO:teuthology.task.internal:Checking packages... 2024-09-12T07:44:49.555 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1' 2024-09-12T07:44:49.555 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-12T07:44:49.555 INFO:teuthology.packaging:ref: None 2024-09-12T07:44:49.555 INFO:teuthology.packaging:tag: None 2024-09-12T07:44:49.556 INFO:teuthology.packaging:branch: wip-dis-testing 2024-09-12T07:44:49.556 INFO:teuthology.packaging:sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:44:49.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2024-09-12T07:44:49.740 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:44:49.742 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-12T07:44:49.748 INFO:teuthology.task.internal:no buildpackages task found 2024-09-12T07:44:49.748 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-12T07:44:49.754 INFO:teuthology.task.internal:Saving configuration 2024-09-12T07:44:49.766 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-12T07:44:49.772 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-12T07:44:49.806 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 07:39:09.287341', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBSiqKwprFI9ZlqqvoS0P6/XWru1cTbl/668yRhiYGJ+'} 2024-09-12T07:44:49.806 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-12T07:44:49.812 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-12T07:44:49.812 INFO:teuthology.run_tasks:Running task console_log... 2024-09-12T07:44:49.853 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1a7a3a3ac0>, signals=[15]) 2024-09-12T07:44:49.853 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-12T07:44:49.859 INFO:teuthology.task.internal:Opening connections... 2024-09-12T07:44:49.859 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T07:44:49.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T07:44:49.949 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-12T07:44:49.956 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-09-12T07:44:50.069 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-09-12T07:44:50.069 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="22.04" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=jammy 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-12T07:44:50.115 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-12T07:44:50.116 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=jammy 2024-09-12T07:44:50.116 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-09-12T07:44:50.148 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-12T07:44:50.158 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-12T07:44:50.164 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-12T07:44:50.164 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-09-12T07:44:50.169 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-12T07:44:50.178 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-12T07:44:50.178 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-09-12T07:44:50.227 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-12T07:44:50.269 INFO:teuthology.run_tasks:Running task kernel... 2024-09-12T07:44:50.285 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-12T07:44:50.285 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-12T07:44:50.285 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T07:44:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:44:50.291 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-12T07:44:50.339 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-56-generic 2024-09-12T07:44:50.340 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.15.0-56-generic 2024-09-12T07:44:50.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2024-09-12T07:44:50.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-12T07:44:50.601 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-12T07:44:50.713 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-12T07:44:50.794 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-12T07:44:50.795 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-12T07:44:50.893 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-12T07:44:50.896 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-12T07:44:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-12T07:44:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-12T07:44:50.899 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-12T07:44:50.919 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-12T07:44:50.921 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-12T07:44:50.921 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-12T07:44:50.927 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-12T07:44:50.931 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-12T07:44:50.932 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-12T07:44:50.933 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-12T07:44:50.933 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-12T07:44:50.933 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-12T07:44:50.968 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-12T07:44:51.150 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-12T07:44:51.289 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-12T07:44:51.483 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-12T07:44:51.640 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-12T07:44:51.654 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-12T07:44:51.655 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-12T07:44:51.722 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-12T07:44:51.724 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-12T07:44:51.735 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-12T07:44:51.736 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-12T07:44:51.760 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-12T07:44:51.791 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-12T07:44:51.797 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-12T07:44:51.805 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-12T07:44:51.806 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-12T07:44:51.807 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-12T07:44:51.880 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-12T07:44:51.881 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-12T07:44:51.882 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-12T07:44:51.884 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-12T07:44:51.884 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-12T07:44:51.886 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-12T07:44:51.887 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-12T07:44:51.968 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-12T07:44:51.969 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-12T07:44:57.183 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 15.8 MB in 2s (7,533 kB/s) 2024-09-12T07:44:58.716 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:44:58.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-12T07:44:58.797 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:44:59.016 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-12T07:44:59.017 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-12T07:44:59.195 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-12T07:44:59.195 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-12T07:44:59.195 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-12T07:44:59.196 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-12T07:44:59.196 INFO:teuthology.orchestra.run.smithi137.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-12T07:44:59.196 INFO:teuthology.orchestra.run.smithi137.stdout: python2.7-minimal 2024-09-12T07:44:59.196 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-12T07:44:59.197 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-09-12T07:44:59.197 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-12T07:44:59.197 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-12T07:44:59.198 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-12T07:44:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-09-12T07:44:59.199 INFO:teuthology.orchestra.run.smithi137.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-12T07:44:59.199 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2024-09-12T07:44:59.199 INFO:teuthology.orchestra.run.smithi137.stdout: thermald 2024-09-12T07:44:59.248 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-09-12T07:44:59.248 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-12T07:44:59.249 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-12T07:44:59.249 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-12T07:44:59.250 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2024-09-12T07:44:59.250 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-12T07:44:59.431 INFO:teuthology.orchestra.run.smithi137.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-12T07:44:59.431 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 113 MB of archives. 2024-09-12T07:44:59.431 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-12T07:44:59.431 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-12T07:45:01.102 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-12T07:45:01.502 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-12T07:45:03.819 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-12T07:45:03.819 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-12T07:45:03.819 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-12T07:45:04.163 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-12T07:45:04.242 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-12T07:45:04.663 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 113 MB in 5s (22.7 MB/s) 2024-09-12T07:45:04.848 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-12T07:45:06.830 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-12T07:45:06.834 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-12T07:45:06.903 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:09.263 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-12T07:45:09.284 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-12T07:45:09.380 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:09.841 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-12T07:45:09.862 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-12T07:45:09.899 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:15.615 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-12T07:45:15.741 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-12T07:45:16.074 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-12T07:45:16.203 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-12T07:45:16.489 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-12T07:45:16.500 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-12T07:45:16.531 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-12T07:45:22.306 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-12T07:45:22.343 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-12T07:45:22.380 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:24.934 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-12T07:45:25.066 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-12T07:45:25.399 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-12T07:45:25.521 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-12T07:45:25.759 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:27.096 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-12T07:45:27.096 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-12T07:45:27.096 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-12T07:45:27.096 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-12T07:45:27.269 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:28.678 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:30.071 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-12T07:45:30.181 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-12T07:45:30.316 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-12T07:45:30.366 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-12T07:45:30.366 INFO:teuthology.orchestra.run.smithi137.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-12T07:45:43.991 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-12T07:45:43.991 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub' 2024-09-12T07:45:44.004 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-12T07:45:44.022 INFO:teuthology.orchestra.run.smithi137.stdout:Generating grub configuration file ... 2024-09-12T07:45:44.344 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-12T07:45:44.372 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-12T07:45:44.736 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-12T07:45:44.742 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-12T07:45:44.806 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-12T07:45:44.811 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-12T07:45:44.963 INFO:teuthology.orchestra.run.smithi137.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-12T07:45:44.963 INFO:teuthology.orchestra.run.smithi137.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-12T07:45:44.963 INFO:teuthology.orchestra.run.smithi137.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-12T07:45:45.012 INFO:teuthology.orchestra.run.smithi137.stdout:done 2024-09-12T07:45:45.520 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 21 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2024-09-12T07:45:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.119.119 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2024-09-12T07:45:45.539 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2024-09-12T07:45:45.539 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-12T07:45:45.539 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-12T07:45:45.539 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-12T07:45:45.540 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-12T07:45:45.540 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-09-12T07:45:45.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2024-09-12T07:45:45.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-12T07:45:45.984 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-12T07:45:45.984 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-12T07:45:46.046 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-12T07:45:46.127 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-12T07:45:47.441 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:45:47.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-12T07:45:47.523 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:45:47.731 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-12T07:45:47.732 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-12T07:45:47.926 INFO:teuthology.orchestra.run.smithi137.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-12T07:45:47.926 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-12T07:45:47.926 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-12T07:45:47.926 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-12T07:45:47.927 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-12T07:45:47.927 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-12T07:45:47.927 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-12T07:45:47.927 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-12T07:45:47.970 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-12T07:45:47.972 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 21 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.119.119 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-12T07:45:47.988 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2024-09-12T07:45:47.989 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2024-09-12T07:45:47.989 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2024-09-12T07:45:47.989 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2024-09-12T07:45:47.989 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-12T07:45:48.034 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.t3vQ5AUUBW 2024-09-12T07:45:48.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub/grub.cfg /tmp/tmp.t3vQ5AUUBW 2024-09-12T07:45:48.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.t3vQ5AUUBW 2024-09-12T07:45:48.203 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.t3vQ5AUUBW is 10KB 2024-09-12T07:45:48.216 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.t3vQ5AUUBW 2024-09-12T07:45:48.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-12T07:45:48.275 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:45:48.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-12T07:45:48.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-12T07:45:48.342 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-12T07:45:48.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo update-grub 2024-09-12T07:45:49.482 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub' 2024-09-12T07:45:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-12T07:45:49.485 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-09-12T07:45:49.736 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-12T07:45:49.742 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-12T07:45:50.052 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-12T07:45:50.058 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-12T07:45:50.098 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-12T07:45:50.103 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-12T07:45:50.218 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-12T07:45:50.218 INFO:teuthology.orchestra.run.smithi137.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-12T07:45:50.218 INFO:teuthology.orchestra.run.smithi137.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-12T07:45:50.239 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-09-12T07:45:50.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-09-12T07:46:20.277 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-12T07:46:20.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:46:20.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T07:46:38.785 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-12T07:46:47.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:46:47.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T07:46:50.877 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-12T07:47:02.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:47:02.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T07:47:21.277 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-12T07:47:36.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:47:36.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T07:47:37.018 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-12T07:47:37.851 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:47:37.851 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-12T07:47:37.852 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-12T07:47:37.899 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-119-generic 2024-09-12T07:47:37.900 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-12T07:47:37.900 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-12T07:47:37.900 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-12T07:47:38.900 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-12T07:47:38.901 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-12T07:47:38.980 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-09-12T07:47:39.013 DEBUG:teuthology.parallel:result is None 2024-09-12T07:47:39.014 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-12T07:47:39.022 INFO:teuthology.task.internal:Creating test directory... 2024-09-12T07:47:39.022 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T07:47:39.028 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-12T07:47:39.034 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-12T07:47:39.040 INFO:teuthology.task.internal:Creating archive directory... 2024-09-12T07:47:39.040 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T07:47:39.091 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-12T07:47:39.100 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-12T07:47:39.100 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T07:47:39.152 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T07:47:39.159 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T07:47:39.161 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-12T07:47:39.168 INFO:teuthology.task.internal:Configuring sudo... 2024-09-12T07:47:39.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T07:47:39.213 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-12T07:47:39.223 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-12T07:47:39.223 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T07:47:39.259 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T07:47:39.308 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T07:47:39.356 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:47:39.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T07:47:39.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-09-12T07:47:39.503 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-12T07:47:39.510 INFO:teuthology.task.internal:Starting timer... 2024-09-12T07:47:39.510 INFO:teuthology.run_tasks:Running task pcp... 2024-09-12T07:47:39.572 INFO:teuthology.run_tasks:Running task selinux... 2024-09-12T07:47:39.580 DEBUG:teuthology.task.selinux:Excluding smithi137: OS 'ubuntu' does not support SELinux 2024-09-12T07:47:39.580 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-12T07:47:39.580 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-12T07:47:39.580 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-12T07:47:39.580 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-12T07:47:39.589 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-12T07:47:39.592 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-12T07:47:39.779 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-12T07:47:39.789 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-12T07:47:39.790 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-12T07:55:07.034 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-09-12T07:55:07.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:55:07.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T07:55:07.130 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-12T07:55:07.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-12T07:55:07.206 INFO:teuthology.run_tasks:Running task clock... 2024-09-12T07:55:07.217 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-12T07:55:07.217 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T07:55:07.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T07:55:07.274 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-12T07:55:07.274 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Command line: ntpd -gq 2024-09-12T07:55:07.275 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: ---------------------------------------------------- 2024-09-12T07:55:07.275 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: ntp-4 is maintained by Network Time Foundation, 2024-09-12T07:55:07.275 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-12T07:55:07.275 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: corporation. Support and training for ntp-4 are 2024-09-12T07:55:07.275 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: available at https://www.nwtime.org/support 2024-09-12T07:55:07.275 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: ---------------------------------------------------- 2024-09-12T07:55:07.276 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: proto: precision = 0.045 usec (-24) 2024-09-12T07:55:07.276 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: basedate set to 2022-02-04 2024-09-12T07:55:07.276 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: gps base set to 2022-02-06 (week 2196) 2024-09-12T07:55:07.276 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-12T07:55:07.276 INFO:teuthology.orchestra.run.smithi137.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-12T07:55:07.276 INFO:teuthology.orchestra.run.smithi137.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-12T07:55:07.277 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: restrict ::: KOD does nothing without LIMITED. 2024-09-12T07:55:07.277 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listen and drop on 0 v6wildcard [::]:123 2024-09-12T07:55:07.278 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-12T07:55:07.278 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listen normally on 2 lo 127.0.0.1:123 2024-09-12T07:55:07.278 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listen normally on 3 enp3s0f1 172.21.15.137:123 2024-09-12T07:55:07.278 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listen normally on 4 lo [::1]:123 2024-09-12T07:55:07.278 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8137%5]:123 2024-09-12T07:55:07.278 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:07 ntpd[18492]: Listening on routing socket on fd #22 for interface updates 2024-09-12T07:55:08.275 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:08.275 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:08.275 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:08.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:08.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:08.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:08.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:08.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:08 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:10.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:10.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:10 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:12.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:12.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:12.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:12.278 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:12.278 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:12.278 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:12.278 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:12.278 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:12 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240912: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-12T07:55:14.276 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240912: Permission denied 2024-09-12T07:55:14.277 INFO:teuthology.orchestra.run.smithi137.stdout:12 Sep 07:55:14 ntpd[18492]: ntpd: time slew -0.001743 s 2024-09-12T07:55:14.277 INFO:teuthology.orchestra.run.smithi137.stdout:ntpd: time slew -0.001743s 2024-09-12T07:55:14.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-12T07:55:14.277 INFO:teuthology.orchestra.run.smithi137.stderr:12 Sep 07:55:14 ntpd[18492]: can't open /var/log/ntpstats/loopstats.20240912: Permission denied 2024-09-12T07:55:14.326 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2024-09-12T07:55:14.326 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2024-09-12T07:55:14.327 INFO:teuthology.orchestra.run.smithi137.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-12T07:55:14.327 INFO:teuthology.orchestra.run.smithi137.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-12T07:55:14.327 INFO:teuthology.orchestra.run.smithi137.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-12T07:55:14.327 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-12T07:55:14.327 INFO:teuthology.run_tasks:Running task install... 2024-09-12T07:55:14.336 DEBUG:teuthology.task.install:project ceph 2024-09-12T07:55:14.336 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}} 2024-09-12T07:55:14.337 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'} 2024-09-12T07:55:14.337 INFO:teuthology.task.install:Using flavor: default 2024-09-12T07:55:14.342 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-12T07:55:14.342 INFO:teuthology.task.install:extra packages: [] 2024-09-12T07:55:14.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-key list | grep Ceph 2024-09-12T07:55:14.422 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-12T07:55:14.450 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-12T07:55:14.450 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph.com (release key) 2024-09-12T07:55:14.451 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-12T07:55:14.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:55:14.622 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default/ 2024-09-12T07:55:14.622 INFO:teuthology.task.install.deb:Package version is 19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:55:14.754 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:55:14.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-12T07:55:14.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-12T07:55:14.941 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-12T07:55:14.998 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy InRelease 2024-09-12T07:55:15.019 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-12T07:55:15.041 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-12T07:55:15.061 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-12T07:55:15.062 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,424 B] 2024-09-12T07:55:15.091 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-12T07:55:15.143 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-12T07:55:15.225 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-12T07:55:15.369 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 29.3 kB in 0s (70.2 kB/s) 2024-09-12T07:55:16.387 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:55:16.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4883-gd61b0886-1jammy cephadm=19.3.0-4883-gd61b0886-1jammy ceph-mds=19.3.0-4883-gd61b0886-1jammy ceph-mgr=19.3.0-4883-gd61b0886-1jammy ceph-common=19.3.0-4883-gd61b0886-1jammy ceph-fuse=19.3.0-4883-gd61b0886-1jammy ceph-test=19.3.0-4883-gd61b0886-1jammy ceph-volume=19.3.0-4883-gd61b0886-1jammy radosgw=19.3.0-4883-gd61b0886-1jammy python3-rados=19.3.0-4883-gd61b0886-1jammy python3-rgw=19.3.0-4883-gd61b0886-1jammy python3-cephfs=19.3.0-4883-gd61b0886-1jammy python3-rbd=19.3.0-4883-gd61b0886-1jammy libcephfs2=19.3.0-4883-gd61b0886-1jammy libcephfs-dev=19.3.0-4883-gd61b0886-1jammy librados2=19.3.0-4883-gd61b0886-1jammy librbd1=19.3.0-4883-gd61b0886-1jammy rbd-fuse=19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:55:16.469 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:55:16.688 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-12T07:55:16.737 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-12T07:55:16.902 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-12T07:55:16.902 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-12T07:55:16.902 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-12T07:55:16.902 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-12T07:55:16.902 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-12T07:55:16.903 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-12T07:55:16.903 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-12T07:55:16.908 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-12T07:55:16.909 INFO:teuthology.orchestra.run.smithi137.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-12T07:55:16.910 INFO:teuthology.orchestra.run.smithi137.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-12T07:55:16.910 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-12T07:55:16.910 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-12T07:55:16.910 INFO:teuthology.orchestra.run.smithi137.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-12T07:55:16.910 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-12T07:55:16.910 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-12T07:55:16.911 INFO:teuthology.orchestra.run.smithi137.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet zip 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-12T07:55:16.912 INFO:teuthology.orchestra.run.smithi137.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-12T07:55:16.913 INFO:teuthology.orchestra.run.smithi137.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-12T07:55:16.913 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2024-09-12T07:55:16.913 INFO:teuthology.orchestra.run.smithi137.stdout: btrfs-tools 2024-09-12T07:55:17.010 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-09-12T07:55:17.010 INFO:teuthology.orchestra.run.smithi137.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-12T07:55:17.010 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-12T07:55:17.010 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-12T07:55:17.010 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-12T07:55:17.010 INFO:teuthology.orchestra.run.smithi137.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-12T07:55:17.011 INFO:teuthology.orchestra.run.smithi137.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-12T07:55:17.012 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-12T07:55:17.013 INFO:teuthology.orchestra.run.smithi137.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-12T07:55:17.160 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-12T07:55:17.161 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 251 MB of archives. 2024-09-12T07:55:17.161 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-12T07:55:17.161 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4883-gd61b0886-1jammy [3,634 kB] 2024-09-12T07:55:17.222 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-12T07:55:17.374 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4883-gd61b0886-1jammy [3,336 kB] 2024-09-12T07:55:17.453 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4883-gd61b0886-1jammy [722 kB] 2024-09-12T07:55:17.468 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4883-gd61b0886-1jammy [356 kB] 2024-09-12T07:55:17.481 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4883-gd61b0886-1jammy [32.8 kB] 2024-09-12T07:55:17.481 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4883-gd61b0886-1jammy [180 kB] 2024-09-12T07:55:17.485 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4883-gd61b0886-1jammy [71.3 kB] 2024-09-12T07:55:17.486 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4883-gd61b0886-1jammy [340 kB] 2024-09-12T07:55:17.492 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4883-gd61b0886-1jammy [7,987 kB] 2024-09-12T07:55:17.642 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4883-gd61b0886-1jammy [112 kB] 2024-09-12T07:55:17.643 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-12T07:55:17.644 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4883-gd61b0886-1jammy [473 kB] 2024-09-12T07:55:17.650 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4883-gd61b0886-1jammy [28.3 MB] 2024-09-12T07:55:17.656 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-12T07:55:17.738 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-12T07:55:17.967 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-12T07:55:17.979 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-12T07:55:18.008 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-12T07:55:18.016 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-12T07:55:18.019 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-12T07:55:18.020 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-12T07:55:18.023 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-12T07:55:18.039 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-12T07:55:18.045 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-12T07:55:18.049 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-12T07:55:18.131 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-12T07:55:18.179 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-12T07:55:18.246 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-12T07:55:18.246 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4883-gd61b0886-1jammy [5,207 kB] 2024-09-12T07:55:18.251 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-12T07:55:18.254 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-12T07:55:18.255 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-12T07:55:18.256 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-12T07:55:18.258 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-12T07:55:18.258 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-12T07:55:18.259 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-12T07:55:18.312 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4883-gd61b0886-1jammy [251 kB] 2024-09-12T07:55:18.314 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4883-gd61b0886-1jammy [124 kB] 2024-09-12T07:55:18.316 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4883-gd61b0886-1jammy [1,562 kB] 2024-09-12T07:55:18.330 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-12T07:55:18.330 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4883-gd61b0886-1jammy [6,309 kB] 2024-09-12T07:55:18.342 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-12T07:55:18.342 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-12T07:55:18.343 INFO:teuthology.orchestra.run.smithi137.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-12T07:55:18.349 INFO:teuthology.orchestra.run.smithi137.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-12T07:55:18.350 INFO:teuthology.orchestra.run.smithi137.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-12T07:55:18.411 INFO:teuthology.orchestra.run.smithi137.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-12T07:55:18.412 INFO:teuthology.orchestra.run.smithi137.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-12T07:55:18.414 INFO:teuthology.orchestra.run.smithi137.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-12T07:55:18.415 INFO:teuthology.orchestra.run.smithi137.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-12T07:55:18.422 INFO:teuthology.orchestra.run.smithi137.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4883-gd61b0886-1jammy [23.1 MB] 2024-09-12T07:55:18.492 INFO:teuthology.orchestra.run.smithi137.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-12T07:55:18.495 INFO:teuthology.orchestra.run.smithi137.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-12T07:55:18.497 INFO:teuthology.orchestra.run.smithi137.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-12T07:55:18.498 INFO:teuthology.orchestra.run.smithi137.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-12T07:55:18.508 INFO:teuthology.orchestra.run.smithi137.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-12T07:55:18.574 INFO:teuthology.orchestra.run.smithi137.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-12T07:55:18.575 INFO:teuthology.orchestra.run.smithi137.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-12T07:55:18.576 INFO:teuthology.orchestra.run.smithi137.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-12T07:55:18.578 INFO:teuthology.orchestra.run.smithi137.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-12T07:55:18.582 INFO:teuthology.orchestra.run.smithi137.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-12T07:55:18.655 INFO:teuthology.orchestra.run.smithi137.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-12T07:55:18.658 INFO:teuthology.orchestra.run.smithi137.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-12T07:55:18.660 INFO:teuthology.orchestra.run.smithi137.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-12T07:55:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-12T07:55:18.666 INFO:teuthology.orchestra.run.smithi137.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-12T07:55:18.737 INFO:teuthology.orchestra.run.smithi137.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-12T07:55:18.742 INFO:teuthology.orchestra.run.smithi137.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-12T07:55:18.874 INFO:teuthology.orchestra.run.smithi137.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4883-gd61b0886-1jammy [14.1 kB] 2024-09-12T07:55:18.874 INFO:teuthology.orchestra.run.smithi137.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4883-gd61b0886-1jammy [880 kB] 2024-09-12T07:55:18.884 INFO:teuthology.orchestra.run.smithi137.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4883-gd61b0886-1jammy [2,454 kB] 2024-09-12T07:55:18.906 INFO:teuthology.orchestra.run.smithi137.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-12T07:55:18.907 INFO:teuthology.orchestra.run.smithi137.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-12T07:55:18.910 INFO:teuthology.orchestra.run.smithi137.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-12T07:55:18.910 INFO:teuthology.orchestra.run.smithi137.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4883-gd61b0886-1jammy [814 kB] 2024-09-12T07:55:18.960 INFO:teuthology.orchestra.run.smithi137.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4883-gd61b0886-1jammy [160 kB] 2024-09-12T07:55:18.961 INFO:teuthology.orchestra.run.smithi137.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-12T07:55:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4883-gd61b0886-1jammy [61.5 MB] 2024-09-12T07:55:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-12T07:55:19.001 INFO:teuthology.orchestra.run.smithi137.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-12T07:55:19.005 INFO:teuthology.orchestra.run.smithi137.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-12T07:55:19.005 INFO:teuthology.orchestra.run.smithi137.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-12T07:55:19.073 INFO:teuthology.orchestra.run.smithi137.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-12T07:55:19.073 INFO:teuthology.orchestra.run.smithi137.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-12T07:55:19.076 INFO:teuthology.orchestra.run.smithi137.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-12T07:55:19.077 INFO:teuthology.orchestra.run.smithi137.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-12T07:55:19.078 INFO:teuthology.orchestra.run.smithi137.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-12T07:55:19.079 INFO:teuthology.orchestra.run.smithi137.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-12T07:55:19.158 INFO:teuthology.orchestra.run.smithi137.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-12T07:55:19.159 INFO:teuthology.orchestra.run.smithi137.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-12T07:55:19.167 INFO:teuthology.orchestra.run.smithi137.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-12T07:55:19.172 INFO:teuthology.orchestra.run.smithi137.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-12T07:55:19.175 INFO:teuthology.orchestra.run.smithi137.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-12T07:55:19.177 INFO:teuthology.orchestra.run.smithi137.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-12T07:55:19.239 INFO:teuthology.orchestra.run.smithi137.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-12T07:55:19.246 INFO:teuthology.orchestra.run.smithi137.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-12T07:55:19.248 INFO:teuthology.orchestra.run.smithi137.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-12T07:55:19.249 INFO:teuthology.orchestra.run.smithi137.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-12T07:55:19.321 INFO:teuthology.orchestra.run.smithi137.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-12T07:55:19.322 INFO:teuthology.orchestra.run.smithi137.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-12T07:55:19.330 INFO:teuthology.orchestra.run.smithi137.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-12T07:55:19.330 INFO:teuthology.orchestra.run.smithi137.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-12T07:55:19.332 INFO:teuthology.orchestra.run.smithi137.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-12T07:55:19.332 INFO:teuthology.orchestra.run.smithi137.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-12T07:55:19.402 INFO:teuthology.orchestra.run.smithi137.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-12T07:55:19.404 INFO:teuthology.orchestra.run.smithi137.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-12T07:55:19.421 INFO:teuthology.orchestra.run.smithi137.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-12T07:55:19.422 INFO:teuthology.orchestra.run.smithi137.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-12T07:55:19.483 INFO:teuthology.orchestra.run.smithi137.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-12T07:55:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-12T07:55:19.490 INFO:teuthology.orchestra.run.smithi137.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-12T07:55:20.243 INFO:teuthology.orchestra.run.smithi137.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4883-gd61b0886-1jammy [8,625 kB] 2024-09-12T07:55:20.362 INFO:teuthology.orchestra.run.smithi137.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4883-gd61b0886-1jammy [14.2 kB] 2024-09-12T07:55:20.362 INFO:teuthology.orchestra.run.smithi137.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4883-gd61b0886-1jammy [55.8 MB] 2024-09-12T07:55:21.521 INFO:teuthology.orchestra.run.smithi137.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4883-gd61b0886-1jammy [130 kB] 2024-09-12T07:55:21.524 INFO:teuthology.orchestra.run.smithi137.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4883-gd61b0886-1jammy [39.0 kB] 2024-09-12T07:55:21.525 INFO:teuthology.orchestra.run.smithi137.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4883-gd61b0886-1jammy [15.9 MB] 2024-09-12T07:55:21.825 INFO:teuthology.orchestra.run.smithi137.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4883-gd61b0886-1jammy [91.2 kB] 2024-09-12T07:55:22.395 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 251 MB in 5s (52.3 MB/s) 2024-09-12T07:55:22.569 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-12T07:55:22.667 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-12T07:55:22.672 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-12T07:55:22.759 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-12T07:55:23.104 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-12T07:55:23.130 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-12T07:55:23.171 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-12T07:55:23.474 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-12T07:55:23.500 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-12T07:55:23.540 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-12T07:55:23.985 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-12T07:55:24.012 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-12T07:55:24.060 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-12T07:55:24.439 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-12T07:55:24.464 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-12T07:55:24.496 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-12T07:55:24.791 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-12T07:55:24.817 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-12T07:55:24.858 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-12T07:55:25.211 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-12T07:55:25.237 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-12T07:55:25.278 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-12T07:55:25.597 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librados2. 2024-09-12T07:55:25.612 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../007-librados2_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:25.647 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librados2 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:26.075 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libnbd0. 2024-09-12T07:55:26.101 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-12T07:55:26.141 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-12T07:55:26.445 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librbd1. 2024-09-12T07:55:26.470 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../009-librbd1_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:26.511 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librbd1 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:26.972 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs2. 2024-09-12T07:55:26.998 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:27.038 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs2 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:27.316 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rados. 2024-09-12T07:55:27.342 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:27.374 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rados (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:27.677 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-12T07:55:27.703 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:55:27.744 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-argparse (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:27.987 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cephfs. 2024-09-12T07:55:28.003 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:28.038 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cephfs (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:28.307 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-12T07:55:28.333 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:55:28.373 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-common (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:28.702 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-12T07:55:28.728 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-12T07:55:28.769 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-12T07:55:29.096 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-prettytable. 2024-09-12T07:55:29.122 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-12T07:55:29.162 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-12T07:55:29.431 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rbd. 2024-09-12T07:55:29.447 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:29.490 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rbd (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:29.760 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-12T07:55:29.775 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-12T07:55:29.810 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-12T07:55:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-12T07:55:30.094 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-12T07:55:30.137 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-12T07:55:30.656 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-12T07:55:30.683 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-12T07:55:30.723 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-12T07:55:31.035 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua5.1. 2024-09-12T07:55:31.061 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-12T07:55:31.101 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-12T07:55:31.404 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-any. 2024-09-12T07:55:31.431 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-12T07:55:31.471 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-12T07:55:31.723 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package zip. 2024-09-12T07:55:31.739 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-12T07:55:31.781 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking zip (3.0-12build2) ... 2024-09-12T07:55:32.110 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package unzip. 2024-09-12T07:55:32.137 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-12T07:55:32.177 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-12T07:55:32.522 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package luarocks. 2024-09-12T07:55:32.548 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-12T07:55:32.580 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-12T07:55:32.979 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librgw2. 2024-09-12T07:55:33.006 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../026-librgw2_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:33.046 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librgw2 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:33.657 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rgw. 2024-09-12T07:55:33.683 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:33.723 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rgw (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:34.060 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-12T07:55:34.086 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-12T07:55:34.126 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-12T07:55:34.387 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libradosstriper1. 2024-09-12T07:55:34.402 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:34.445 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libradosstriper1 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:34.866 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-common. 2024-09-12T07:55:34.892 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:34.932 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-common (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:36.387 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-base. 2024-09-12T07:55:36.414 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:36.463 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-base (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:36.999 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-12T07:55:37.015 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-12T07:55:37.049 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-12T07:55:37.344 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cheroot. 2024-09-12T07:55:37.371 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-12T07:55:37.410 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-12T07:55:37.739 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-12T07:55:37.765 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-12T07:55:37.805 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-12T07:55:38.058 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-12T07:55:38.085 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-12T07:55:38.116 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-12T07:55:38.412 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-12T07:55:38.438 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-12T07:55:38.477 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-12T07:55:38.780 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempora. 2024-09-12T07:55:38.807 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-12T07:55:38.847 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-12T07:55:39.124 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-portend. 2024-09-12T07:55:39.151 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-12T07:55:39.182 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-12T07:55:39.444 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-12T07:55:39.470 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-12T07:55:39.510 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-12T07:55:39.838 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-12T07:55:39.865 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-12T07:55:39.905 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-12T07:55:40.192 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-natsort. 2024-09-12T07:55:40.218 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-12T07:55:40.250 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-12T07:55:40.563 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-logutils. 2024-09-12T07:55:40.590 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-12T07:55:40.630 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-12T07:55:41.116 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-12T07:55:41.143 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-12T07:55:41.182 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-12T07:55:41.485 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-mako. 2024-09-12T07:55:41.512 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-12T07:55:41.552 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-12T07:55:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-12T07:55:41.907 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-12T07:55:41.947 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-12T07:55:42.225 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-12T07:55:42.252 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-12T07:55:42.291 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-12T07:55:42.544 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webob. 2024-09-12T07:55:42.570 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-12T07:55:42.610 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-12T07:55:42.955 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-waitress. 2024-09-12T07:55:42.982 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-12T07:55:43.025 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-12T07:55:43.316 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempita. 2024-09-12T07:55:43.343 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-12T07:55:43.408 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-12T07:55:43.727 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-paste. 2024-09-12T07:55:43.754 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-12T07:55:43.794 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-12T07:55:44.114 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-12T07:55:44.130 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-12T07:55:44.172 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-12T07:55:44.475 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-12T07:55:44.501 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-12T07:55:44.541 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-12T07:55:44.827 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webtest. 2024-09-12T07:55:44.854 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-12T07:55:44.886 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-12T07:55:45.172 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pecan. 2024-09-12T07:55:45.199 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-12T07:55:45.238 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-12T07:55:45.560 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-certifi. 2024-09-12T07:55:45.587 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-12T07:55:45.618 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-12T07:55:45.930 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-idna. 2024-09-12T07:55:45.957 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-12T07:55:45.996 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-12T07:55:46.307 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-urllib3. 2024-09-12T07:55:46.334 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-12T07:55:46.373 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-12T07:55:46.718 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests. 2024-09-12T07:55:46.745 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-12T07:55:46.785 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-12T07:55:47.130 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-12T07:55:47.157 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-12T07:55:47.196 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-12T07:55:47.491 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-12T07:55:47.506 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:55:47.549 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:47.904 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-12T07:55:47.931 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:47.970 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:48.273 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-12T07:55:48.300 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-12T07:55:48.339 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-12T07:55:48.644 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-openssl. 2024-09-12T07:55:48.669 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-12T07:55:48.709 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-12T07:55:48.995 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr. 2024-09-12T07:55:49.019 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:49.053 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:49.457 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mon. 2024-09-12T07:55:49.483 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:49.523 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mon (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:50.135 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-osd. 2024-09-12T07:55:50.162 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:50.201 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-osd (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:51.855 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph. 2024-09-12T07:55:51.870 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../067-ceph_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:51.913 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:52.207 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-fuse. 2024-09-12T07:55:52.234 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:52.265 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-fuse (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:52.610 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mds. 2024-09-12T07:55:52.636 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:52.676 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mds (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:53.079 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cephadm. 2024-09-12T07:55:53.106 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../070-cephadm_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:55:53.146 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cephadm (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:53.457 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-12T07:55:53.484 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-12T07:55:53.523 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-12T07:55:53.818 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-12T07:55:53.845 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-12T07:55:53.884 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-12T07:55:54.931 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-babel. 2024-09-12T07:55:54.958 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-12T07:55:54.997 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-12T07:55:55.351 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jinja2. 2024-09-12T07:55:55.377 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-12T07:55:55.417 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-12T07:55:55.762 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-12T07:55:55.788 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:55:55.828 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:56.165 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-12T07:55:56.192 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-12T07:55:56.231 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-12T07:55:56.526 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-routes. 2024-09-12T07:55:56.541 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-12T07:55:56.576 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-12T07:55:56.887 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-12T07:55:56.914 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:55:56.962 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:55:59.500 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-12T07:55:59.517 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-12T07:55:59.558 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-12T07:56:00.045 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-joblib. 2024-09-12T07:56:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-12T07:56:00.111 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-12T07:56:00.490 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-12T07:56:00.518 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-12T07:56:00.557 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-12T07:56:00.859 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn. 2024-09-12T07:56:00.887 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-12T07:56:01.011 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-12T07:56:01.870 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-12T07:56:01.887 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:56:01.920 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:02.873 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cachetools. 2024-09-12T07:56:02.901 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-12T07:56:02.940 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-12T07:56:03.243 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-12T07:56:03.271 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-12T07:56:03.309 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-12T07:56:03.654 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-12T07:56:03.682 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-12T07:56:03.721 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-12T07:56:04.032 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rsa. 2024-09-12T07:56:04.060 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-12T07:56:04.098 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-12T07:56:04.419 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-google-auth. 2024-09-12T07:56:04.446 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-12T07:56:04.485 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-12T07:56:04.797 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-12T07:56:04.828 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-12T07:56:04.888 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-12T07:56:05.200 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-websocket. 2024-09-12T07:56:05.228 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-12T07:56:05.266 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-12T07:56:05.578 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-12T07:56:05.606 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-12T07:56:05.644 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-12T07:56:06.300 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-12T07:56:06.316 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:56:06.358 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:06.669 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-12T07:56:06.697 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-12T07:56:06.735 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-12T07:56:07.039 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-12T07:56:07.067 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-12T07:56:07.097 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-12T07:56:07.358 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package jq. 2024-09-12T07:56:07.386 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-12T07:56:07.424 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-12T07:56:07.678 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package socat. 2024-09-12T07:56:07.705 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-12T07:56:07.744 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-12T07:56:08.064 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package xmlstarlet. 2024-09-12T07:56:08.092 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-12T07:56:08.130 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-12T07:56:08.518 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-test. 2024-09-12T07:56:08.546 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:56:08.584 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-test (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:11.005 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-volume. 2024-09-12T07:56:11.033 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4883-gd61b0886-1jammy_all.deb ... 2024-09-12T07:56:11.088 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-volume (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:11.367 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-12T07:56:11.383 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:56:11.416 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs-dev (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:11.703 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-12T07:56:11.731 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-12T07:56:11.818 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-12T07:56:12.216 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-12T07:56:12.244 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-12T07:56:12.282 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-12T07:56:12.594 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package nvme-cli. 2024-09-12T07:56:12.622 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-12T07:56:12.652 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-12T07:56:12.991 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package pkg-config. 2024-09-12T07:56:13.019 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-12T07:56:13.057 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-12T07:56:13.310 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-12T07:56:13.326 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-12T07:56:13.369 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-12T07:56:13.789 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-12T07:56:13.817 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-12T07:56:13.855 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-12T07:56:14.124 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastescript. 2024-09-12T07:56:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-12T07:56:14.174 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-12T07:56:14.469 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pluggy. 2024-09-12T07:56:14.498 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-12T07:56:14.535 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-12T07:56:14.847 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-psutil. 2024-09-12T07:56:14.876 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-12T07:56:14.913 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-12T07:56:15.241 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-py. 2024-09-12T07:56:15.270 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-12T07:56:15.308 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-12T07:56:15.661 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pygments. 2024-09-12T07:56:15.690 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-12T07:56:15.727 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-12T07:56:16.173 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-12T07:56:16.202 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-12T07:56:16.239 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-12T07:56:16.526 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-toml. 2024-09-12T07:56:16.554 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-12T07:56:16.592 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-12T07:56:16.853 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pytest. 2024-09-12T07:56:16.870 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-12T07:56:16.911 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-12T07:56:17.223 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplejson. 2024-09-12T07:56:17.252 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-12T07:56:17.290 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-12T07:56:17.567 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-12T07:56:17.596 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-12T07:56:17.625 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-12T07:56:18.338 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package radosgw. 2024-09-12T07:56:18.366 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../117-radosgw_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:56:18.395 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking radosgw (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:19.290 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package rbd-fuse. 2024-09-12T07:56:19.307 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4883-gd61b0886-1jammy_amd64.deb ... 2024-09-12T07:56:19.349 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking rbd-fuse (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:19.645 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-12T07:56:19.928 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-12T07:56:20.032 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-12T07:56:20.295 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-12T07:56:20.654 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-12T07:56:21.077 INFO:teuthology.orchestra.run.smithi137.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-12T07:56:21.133 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cephadm (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:21.674 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user cephadm....done 2024-09-12T07:56:21.797 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-12T07:56:22.074 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-12T07:56:22.592 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-12T07:56:22.718 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-12T07:56:22.994 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-12T07:56:23.253 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-12T07:56:23.379 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-12T07:56:23.714 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-12T07:56:24.075 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-12T07:56:24.326 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-12T07:56:24.610 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-12T07:56:24.845 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-12T07:56:25.096 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-12T07:56:25.412 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-12T07:56:25.707 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-argparse (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:25.974 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-12T07:56:26.100 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-12T07:56:26.226 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-12T07:56:26.372 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-12T07:56:26.503 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-12T07:56:26.754 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-12T07:56:27.160 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-12T07:56:27.194 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-12T07:56:27.249 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-12T07:56:27.358 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-12T07:56:27.718 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-12T07:56:27.844 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-12T07:56:28.112 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-12T07:56:28.371 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-12T07:56:28.681 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up zip (3.0-12build2) ... 2024-09-12T07:56:28.790 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-12T07:56:29.450 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-12T07:56:29.718 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-12T07:56:29.844 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-12T07:56:29.987 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-12T07:56:30.271 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-12T07:56:30.655 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-12T07:56:30.915 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-12T07:56:31.325 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-12T07:56:31.618 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-12T07:56:31.961 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-12T07:56:32.212 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-12T07:56:32.313 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-common (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:32.597 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-12T07:56:32.898 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-12T07:56:33.996 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-12T07:56:34.231 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-12T07:56:34.340 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-12T07:56:34.591 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-12T07:56:34.734 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-12T07:56:34.860 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-12T07:56:35.153 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-12T07:56:35.404 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-12T07:56:35.646 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-12T07:56:35.756 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-12T07:56:36.041 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-12T07:56:36.292 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-12T07:56:36.551 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-12T07:56:36.810 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-12T07:56:37.061 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-12T07:56:37.337 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-12T07:56:37.463 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-12T07:56:37.740 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-12T07:56:37.866 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-12T07:56:38.159 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-12T07:56:38.468 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-12T07:56:38.728 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-12T07:56:38.845 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-12T07:56:38.946 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-12T07:56:39.021 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-12T07:56:39.290 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-12T07:56:39.391 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-12T07:56:39.767 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-12T07:56:40.026 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-12T07:56:40.152 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-12T07:56:40.420 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-12T07:56:40.521 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-12T07:56:40.822 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-12T07:56:41.115 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-12T07:56:41.391 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-12T07:56:41.517 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-12T07:56:41.776 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-12T07:56:42.154 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-12T07:56:42.414 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-12T07:56:42.816 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-12T07:56:43.101 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-12T07:56:43.436 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-12T07:56:43.537 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librados2 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:43.638 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:43.748 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-12T07:56:44.084 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs2 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:44.210 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libradosstriper1 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:44.353 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-12T07:56:44.629 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librbd1 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:44.738 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-modules-core (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:44.840 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-fuse (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:45.035 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-12T07:56:45.035 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-12T07:56:45.507 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs-dev (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:45.633 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rados (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:45.759 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librgw2 (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:45.885 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-12T07:56:47.097 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rbd (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:47.198 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up rbd-fuse (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:47.341 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rgw (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:47.467 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cephfs (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:47.593 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-common (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:48.144 INFO:teuthology.orchestra.run.smithi137.stdout:Adding group ceph....done 2024-09-12T07:56:48.479 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user ceph....done 2024-09-12T07:56:48.647 INFO:teuthology.orchestra.run.smithi137.stdout:Setting system user ceph properties....done 2024-09-12T07:56:48.664 INFO:teuthology.orchestra.run.smithi137.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-12T07:56:48.799 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-12T07:56:49.104 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-12T07:56:49.577 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-test (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:49.686 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up radosgw (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:50.135 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-12T07:56:50.135 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-12T07:56:50.588 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-base (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:50.947 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-12T07:56:51.432 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mds (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:51.695 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-12T07:56:51.695 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-12T07:56:52.217 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:52.433 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-12T07:56:52.433 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-12T07:56:52.860 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-osd (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:53.123 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-12T07:56:53.123 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-12T07:56:53.587 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:53.688 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:53.822 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mon (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:54.033 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-12T07:56:54.033 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-12T07:56:54.507 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-cephadm (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:54.649 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:54.776 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-dashboard (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:54.927 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-volume (19.3.0-4883-gd61b0886-1jammy) ... 2024-09-12T07:56:55.289 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-12T07:56:55.388 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-12T07:56:55.516 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-12T07:56:56.658 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-12T07:56:57.294 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-12T07:56:57.301 DEBUG:teuthology.parallel:result is None 2024-09-12T07:56:57.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:56:57.463 DEBUG:teuthology.orchestra.run.smithi137:> dpkg-query -W -f '${Version}' ceph 2024-09-12T07:56:57.481 INFO:teuthology.orchestra.run.smithi137.stdout:19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:56:57.482 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:56:57.482 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883-gd61b0886-1jammy is installed. 2024-09-12T07:56:57.484 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-12T07:56:57.484 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:57.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T07:56:57.538 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-12T07:56:57.539 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:57.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T07:56:57.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T07:56:57.649 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-12T07:56:57.649 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:57.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T07:56:57.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T07:56:57.756 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-12T07:56:57.757 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:57.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T07:56:57.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T07:56:57.868 INFO:teuthology.run_tasks:Running task ceph... 2024-09-12T07:56:57.952 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-12T07:56:57.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-09-12T07:56:57.965 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-12T07:56:57.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-12T07:56:58.021 INFO:tasks.ceph:Creating extra log directories... 2024-09-12T07:56:58.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-12T07:56:58.084 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-12T07:56:58.084 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-12T07:56:58.084 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438', 'branch': 'wip-dis-testing', 'description': 'rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7902438', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1680, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'sleep_before_teardown': 0, 'subset': '42/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'targets': {'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFVWz4mh4Jf0nGAQ9Jc69Q5d75SVGnKGVSM/5drhDnc1kvuIxWsMRRm9EyLVTEOLSKNDiaibvqbIbHIB2licWtU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-12_04:46:10', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-12T07:56:58.085 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-12T07:56:58.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T07:56:58.182 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:58.182 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T07:56:58.230 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T07:56:58.230 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-12 07:51:35.419392466 +0000 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-12 07:51:35.179394611 +0000 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-12 07:51:35.179394611 +0000 2024-09-12T07:56:58.278 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-12T07:56:58.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T07:56:58.332 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-12T07:56:58.332 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-12T07:56:58.332 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000438303 s, 1.2 MB/s 2024-09-12T07:56:58.333 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T07:56:58.384 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-12 07:51:35.879388358 +0000 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-12 07:51:35.663390287 +0000 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-12 07:51:35.663390287 +0000 2024-09-12T07:56:58.430 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-12T07:56:58.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T07:56:58.484 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-12T07:56:58.484 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-12T07:56:58.484 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000348626 s, 1.5 MB/s 2024-09-12T07:56:58.486 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T07:56:58.539 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-09-12T07:56:58.586 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T07:56:58.586 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T07:56:58.586 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-12T07:56:58.586 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T07:56:58.586 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-12 07:51:36.359384070 +0000 2024-09-12T07:56:58.587 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-12 07:51:36.103386357 +0000 2024-09-12T07:56:58.587 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-12 07:51:36.103386357 +0000 2024-09-12T07:56:58.587 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-12T07:56:58.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T07:56:58.639 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-12T07:56:58.640 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-12T07:56:58.640 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000340075 s, 1.5 MB/s 2024-09-12T07:56:58.641 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T07:56:58.691 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 940 Links: 1 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-12 07:51:36.847379712 +0000 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-12 07:51:36.587382034 +0000 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-12 07:51:36.587382034 +0000 2024-09-12T07:56:58.738 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-12T07:56:58.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T07:56:58.792 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-12T07:56:58.792 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-12T07:56:58.792 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000287988 s, 1.8 MB/s 2024-09-12T07:56:58.793 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T07:56:58.843 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T07:56:58.844 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T07:56:58.844 INFO:tasks.ceph:Generating config... 2024-09-12T07:56:58.845 INFO:tasks.ceph:[client] rbd default data pool = datapool 2024-09-12T07:56:58.845 INFO:tasks.ceph:[client] rbd default features = 61 2024-09-12T07:56:58.845 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-12T07:56:58.845 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-12T07:56:58.845 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-12T07:56:58.845 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-12T07:56:58.845 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-12T07:56:58.846 INFO:tasks.ceph:[mgr] debug rbd = 20 2024-09-12T07:56:58.846 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-12T07:56:58.846 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-12T07:56:58.846 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-12T07:56:58.846 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-12T07:56:58.847 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-12T07:56:58.847 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-12T07:56:58.847 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-12T07:56:58.847 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-12T07:56:58.847 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-12T07:56:58.847 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-12T07:56:58.847 INFO:tasks.ceph:Setting up mon.a... 2024-09-12T07:56:58.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-12T07:56:58.918 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.keyring 2024-09-12T07:56:58.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-12T07:56:59.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-12T07:56:59.069 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.137')] 2024-09-12T07:56:59.070 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.137', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 61}, 'mon.a': {}} 2024-09-12T07:56:59.070 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:59.070 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T07:56:59.117 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.137 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:56:59.197 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: generated fsid b2d66243-42d9-4ed0-b04a-17c8c73580cd 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = quincy 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 0 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:fsid b2d66243-42d9-4ed0-b04a-17c8c73580cd 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:last_changed 2024-09-12T07:56:59.199038+0000 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-09-12T07:56:59.199038+0000 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:min_mon_release 17 (quincy) 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:election_strategy: 1 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.a 2024-09-12T07:56:59.198 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-12T07:56:59.199 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T07:56:59.246 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b2d66243-42d9-4ed0-b04a-17c8c73580cd... 2024-09-12T07:56:59.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-09-12T07:56:59.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-09-12T07:56:59.325 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-09-12T07:56:59.326 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.137 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: mon client directed command retry = 5 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on pool no app = False 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: ms inject socket failures = 5000 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = b2d66243-42d9-4ed0-b04a-17c8c73580cd 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-09-12T07:56:59.327 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-09-12T07:56:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: bdev async discard = True 2024-09-12T07:56:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: bdev enable discard = True 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore allocator = stupid 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore block size = 96636764160 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore fsck on mount = True 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 1/20 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 1/20 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 4/10 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T07:56:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd full ratio = 0.9 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd nearfull ratio = 0.8 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: osd failsafe full ratio = 0.95 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = bluestore 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: osd shutdown pgref assert = True 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-09-12T07:56:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: debug rbd = 20 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-09-12T07:56:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:56:59.333 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: rbd default data pool = datapool 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout: rbd default features = 61 2024-09-12T07:56:59.334 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-09-12T07:56:59.335 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-12T07:56:59.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-12T07:56:59.425 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-12T07:56:59.425 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:59.425 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-12T07:56:59.430 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:59.430 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-12T07:56:59.477 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T07:56:59.477 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:59.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-12T07:56:59.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-12T07:56:59.542 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:56:59.542 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:56:59.589 INFO:tasks.ceph:Setting up mon nodes... 2024-09-12T07:56:59.590 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-12T07:56:59.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-12T07:56:59.675 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-12T07:56:59.677 INFO:tasks.ceph:Setting up mds nodes... 2024-09-12T07:56:59.678 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-12T07:56:59.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-12T07:56:59.753 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-12T07:56:59.763 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-12T07:56:59.764 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T07:56:59.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-12T07:56:59.816 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T07:56:59.816 INFO:tasks.ceph:role: osd.0 2024-09-12T07:56:59.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T07:56:59.816 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-12T07:56:59.876 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T07:56:59.877 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T07:57:00.347 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-12T07:57:00.348 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-12T07:57:00.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-12T07:57:00.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-12T07:57:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:57:00.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-12T07:57:00.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-12T07:57:00.511 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T07:57:00.511 INFO:tasks.ceph:role: osd.1 2024-09-12T07:57:00.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T07:57:00.511 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-12T07:57:00.574 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T07:57:00.574 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T07:57:00.574 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T07:57:00.574 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-12T07:57:00.574 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T07:57:00.575 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-12T07:57:00.575 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T07:57:00.575 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-12T07:57:00.575 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T07:57:00.575 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T07:57:01.035 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-12T07:57:01.037 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-12T07:57:01.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-12T07:57:01.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-12T07:57:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:57:01.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-12T07:57:01.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-12T07:57:01.261 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T07:57:01.261 INFO:tasks.ceph:role: osd.2 2024-09-12T07:57:01.261 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T07:57:01.261 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-12T07:57:01.325 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T07:57:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T07:57:01.326 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T07:57:01.803 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-12T07:57:01.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-12T07:57:01.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-12T07:57:01.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-12T07:57:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:57:01.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-12T07:57:01.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:57:02.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:02.089+0000 7eff26cec7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-12T07:57:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:02.089+0000 7eff26cec7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-12T07:57:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:02.089+0000 7eff26cec7c0 -1 bdev(0x55cf3f8f1800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-12T07:57:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:02.089+0000 7eff26cec7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-12T07:57:05.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-12T07:57:05.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:57:05.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:05.313+0000 7f9ec308f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-12T07:57:05.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:05.313+0000 7f9ec308f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-12T07:57:05.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:05.313+0000 7f9ec308f7c0 -1 bdev(0x5619a2839800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-12T07:57:05.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:05.313+0000 7f9ec308f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-12T07:57:08.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-12T07:57:08.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:57:08.587 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:08.585+0000 7fc10f0917c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-12T07:57:08.588 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:08.585+0000 7fc10f0917c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-12T07:57:08.588 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:08.585+0000 7fc10f0917c0 -1 bdev(0x55edb906f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-12T07:57:08.588 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T07:57:08.585+0000 7fc10f0917c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-12T07:57:11.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-12T07:57:11.785 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-12T07:57:11.785 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:11.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-12T07:57:11.845 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:11.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-12T07:57:11.900 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:11.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-12T07:57:11.953 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:11.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-12T07:57:12.008 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:12.008 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-12T07:57:12.054 INFO:tasks.ceph:Adding keys to all mons... 2024-09-12T07:57:12.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-12T07:57:12.107 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-09-12T07:57:12.108 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDLnuJm/z1TKBAABzhAsu2YwoZn1ieYsyEYaQ== 2024-09-12T07:57:12.108 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-09-12T07:57:12.108 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDOnuJmxq+SBRAA0QCWpG7wt5QWmYLoSMOqWw== 2024-09-12T07:57:12.108 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-09-12T07:57:12.108 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDRnuJml8fmEhAAKvr1EnNGJCrLyccuG7kCfg== 2024-09-12T07:57:12.108 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-09-12T07:57:12.109 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDUnuJmv2wfIxAAMk0klhFcQDbnXmL5x8IS4A== 2024-09-12T07:57:12.109 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-09-12T07:57:12.109 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDLnuJmjbz9LBAA8xq9E8Gdibse9FwNfu5FyQ== 2024-09-12T07:57:12.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-12T07:57:12.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T07:57:12.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T07:57:12.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T07:57:12.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T07:57:12.548 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-12T07:57:12.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-12T07:57:12.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-12T07:57:12.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-12T07:57:12.711 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-12T07:57:12.757 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-12T07:57:12.758 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-12T07:57:12.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-12T07:57:12.806 INFO:tasks.ceph.mon.a:Started 2024-09-12T07:57:12.807 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-12T07:57:12.807 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-12T07:57:12.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-12T07:57:12.809 INFO:tasks.ceph.mgr.x:Started 2024-09-12T07:57:12.809 DEBUG:tasks.ceph:set 0 configs 2024-09-12T07:57:12.809 INFO:tasks.ceph:Setting crush tunables to default 2024-09-12T07:57:12.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd crush tunables default 2024-09-12T07:57:13.066 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-09-12T07:57:13.084 INFO:tasks.ceph:check_enable_crimson: False 2024-09-12T07:57:13.084 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-12T07:57:13.084 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:13.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-12T07:57:13.095 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:13.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-12T07:57:13.148 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:13.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-12T07:57:13.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new d7f68ec7-0a8d-4dca-b021-98e1568bcbc8 0 2024-09-12T07:57:13.449 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-09-12T07:57:13.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new b2f260db-e8a7-4a95-98c3-0b180d2c805e 1 2024-09-12T07:57:13.666 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-09-12T07:57:13.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 8a2900b6-32d2-4e1d-99be-dc483c64abfc 2 2024-09-12T07:57:13.897 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-09-12T07:57:13.915 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-12T07:57:13.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-12T07:57:13.917 INFO:tasks.ceph.osd.0:Started 2024-09-12T07:57:13.917 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-12T07:57:13.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-12T07:57:13.919 INFO:tasks.ceph.osd.1:Started 2024-09-12T07:57:13.920 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-12T07:57:13.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-12T07:57:13.922 INFO:tasks.ceph.osd.2:Started 2024-09-12T07:57:13.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-12T07:57:14.106 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:14.106 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":5,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T07:57:13.898448+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T07:57:14.122 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-12T07:57:14.122 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-12T07:57:14.424 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-12T07:57:14.603 INFO:teuthology.misc.health.smithi137.stdout: 2024-09-12T07:57:14.603 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":5,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T07:57:13.898448+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T07:57:14.618 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T07:57:15.122 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T07:57:15.121+0000 7f2a69d017c0 -1 Falling back to public interface 2024-09-12T07:57:15.355 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-12T07:57:15.353+0000 7f4f677fe7c0 -1 Falling back to public interface 2024-09-12T07:57:15.846 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:57:15.845+0000 7fb1757647c0 -1 Falling back to public interface 2024-09-12T07:57:15.946 INFO:tasks.ceph.mgr.x.smithi137.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-12T07:57:15.946 INFO:tasks.ceph.mgr.x.smithi137.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-12T07:57:15.946 INFO:tasks.ceph.mgr.x.smithi137.stderr: from numpy import show_config as show_numpy_config 2024-09-12T07:57:18.866 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T07:57:18.865+0000 7f9d33c38640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-12T07:57:19.344 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T07:57:19.341+0000 7f2a69d017c0 -1 osd.0 0 log_to_monitors true 2024-09-12T07:57:19.444 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-12T07:57:19.441+0000 7f4f677fe7c0 -1 osd.2 0 log_to_monitors true 2024-09-12T07:57:20.161 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:57:20.161+0000 7fb1757647c0 -1 osd.1 0 log_to_monitors true 2024-09-12T07:57:20.921 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-12T07:57:21.274 INFO:teuthology.misc.health.smithi137.stdout: 2024-09-12T07:57:21.274 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":7,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T07:57:20.913628+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T07:57:21.294 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T07:57:22.542 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T07:57:22.541+0000 7f2a65caa640 -1 osd.0 0 waiting for initial osdmap 2024-09-12T07:57:22.571 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T07:57:22.569+0000 7f2a5dadc640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T07:57:22.802 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-12T07:57:22.801+0000 7f4f637a7640 -1 osd.2 0 waiting for initial osdmap 2024-09-12T07:57:22.831 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-12T07:57:22.829+0000 7f4f5b5d9640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T07:57:23.106 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:57:23.105+0000 7fb17170d640 -1 osd.1 0 waiting for initial osdmap 2024-09-12T07:57:23.146 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:57:23.145+0000 7fb16953f640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T07:57:27.597 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-12T07:57:27.908 INFO:teuthology.misc.health.smithi137.stdout: 2024-09-12T07:57:27.908 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":12,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T07:57:26.549455+0000","last_up_change":"2024-09-12T07:57:23.545318+0000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T07:57:23.901045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6801","nonce":998099114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6803","nonce":998099114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6807","nonce":998099114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6805","nonce":998099114}]},"public_addr":"172.21.15.137:6801/998099114","cluster_addr":"172.21.15.137:6803/998099114","heartbeat_back_addr":"172.21.15.137:6807/998099114","heartbeat_front_addr":"172.21.15.137:6805/998099114","state":["exists","up"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6817","nonce":547903978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6819","nonce":547903978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6823","nonce":547903978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6821","nonce":547903978}]},"public_addr":"172.21.15.137:6817/547903978","cluster_addr":"172.21.15.137:6819/547903978","heartbeat_back_addr":"172.21.15.137:6823/547903978","heartbeat_front_addr":"172.21.15.137:6821/547903978","state":["exists","up"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6809","nonce":99608485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6811","nonce":99608485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6815","nonce":99608485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6813","nonce":99608485}]},"public_addr":"172.21.15.137:6809/99608485","cluster_addr":"172.21.15.137:6811/99608485","heartbeat_back_addr":"172.21.15.137:6815/99608485","heartbeat_front_addr":"172.21.15.137:6813/99608485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.352359+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:21.139340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.401170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T07:57:27.925 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-12T07:57:27.926 INFO:tasks.ceph:Creating RBD pool 2024-09-12T07:57:27.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-12T07:57:28.559 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'rbd' created 2024-09-12T07:57:28.586 DEBUG:teuthology.orchestra.run.smithi137:> rbd --cluster ceph pool init rbd 2024-09-12T07:57:31.603 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-12T07:57:31.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-12T07:57:31.604 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-12T07:57:31.917 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:31.935 INFO:teuthology.orchestra.run.smithi137.stdout:[{"version":4,"timestamp":"2024-09-12T07:57:23.108599+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"4976.027102"}]}] 2024-09-12T07:57:31.935 INFO:tasks.ceph_manager:config epoch is 4 2024-09-12T07:57:31.935 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-12T07:57:31.935 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-12T07:57:31.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-12T07:57:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:32.284 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3076968402},{"type":"v1","addr":"172.21.15.137:6825","nonce":3076968402}]},"active_addr":"172.21.15.137:6825/3076968402","active_change":"2024-09-12T07:57:17.852704+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1311542896}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":4910983}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1667139831}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3653552997}]}]} 2024-09-12T07:57:32.288 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-12T07:57:32.289 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-12T07:57:32.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-12T07:57:32.593 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:32.593 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":16,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T07:57:31.571995+0000","last_up_change":"2024-09-12T07:57:23.545318+0000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T07:57:23.901045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T07:57:28.240312+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6801","nonce":998099114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6803","nonce":998099114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6807","nonce":998099114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6805","nonce":998099114}]},"public_addr":"172.21.15.137:6801/998099114","cluster_addr":"172.21.15.137:6803/998099114","heartbeat_back_addr":"172.21.15.137:6807/998099114","heartbeat_front_addr":"172.21.15.137:6805/998099114","state":["exists","up"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6817","nonce":547903978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6819","nonce":547903978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6823","nonce":547903978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6821","nonce":547903978}]},"public_addr":"172.21.15.137:6817/547903978","cluster_addr":"172.21.15.137:6819/547903978","heartbeat_back_addr":"172.21.15.137:6823/547903978","heartbeat_front_addr":"172.21.15.137:6821/547903978","state":["exists","up"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6809","nonce":99608485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6811","nonce":99608485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6815","nonce":99608485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6813","nonce":99608485}]},"public_addr":"172.21.15.137:6809/99608485","cluster_addr":"172.21.15.137:6811/99608485","heartbeat_back_addr":"172.21.15.137:6815/99608485","heartbeat_front_addr":"172.21.15.137:6813/99608485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.352359+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:21.139340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.401170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T07:57:32.611 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-12T07:57:32.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-12T07:57:32.899 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:32.900 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":16,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T07:57:31.571995+0000","last_up_change":"2024-09-12T07:57:23.545318+0000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T07:57:23.901045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T07:57:28.240312+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6801","nonce":998099114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6803","nonce":998099114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6807","nonce":998099114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6805","nonce":998099114}]},"public_addr":"172.21.15.137:6801/998099114","cluster_addr":"172.21.15.137:6803/998099114","heartbeat_back_addr":"172.21.15.137:6807/998099114","heartbeat_front_addr":"172.21.15.137:6805/998099114","state":["exists","up"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6817","nonce":547903978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6819","nonce":547903978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6823","nonce":547903978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6821","nonce":547903978}]},"public_addr":"172.21.15.137:6817/547903978","cluster_addr":"172.21.15.137:6819/547903978","heartbeat_back_addr":"172.21.15.137:6823/547903978","heartbeat_front_addr":"172.21.15.137:6821/547903978","state":["exists","up"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6809","nonce":99608485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6811","nonce":99608485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6815","nonce":99608485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6813","nonce":99608485}]},"public_addr":"172.21.15.137:6809/99608485","cluster_addr":"172.21.15.137:6811/99608485","heartbeat_back_addr":"172.21.15.137:6815/99608485","heartbeat_front_addr":"172.21.15.137:6813/99608485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.352359+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:21.139340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.401170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T07:57:32.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-12T07:57:32.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-12T07:57:32.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-12T07:57:33.068 INFO:teuthology.orchestra.run.smithi137.stdout:38654705668 2024-09-12T07:57:33.069 INFO:teuthology.orchestra.run.smithi137.stdout:38654705670 2024-09-12T07:57:33.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-12T07:57:33.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-12T07:57:33.073 INFO:teuthology.orchestra.run.smithi137.stdout:38654705669 2024-09-12T07:57:33.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-12T07:57:33.356 INFO:teuthology.orchestra.run.smithi137.stdout:38654705667 2024-09-12T07:57:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:38654705669 2024-09-12T07:57:33.377 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2024-09-12T07:57:33.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-09-12T07:57:33.408 INFO:teuthology.orchestra.run.smithi137.stdout:38654705668 2024-09-12T07:57:33.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2024-09-12T07:57:34.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-12T07:57:34.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-12T07:57:34.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-12T07:57:34.694 INFO:teuthology.orchestra.run.smithi137.stdout:38654705668 2024-09-12T07:57:34.710 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-09-12T07:57:34.710 DEBUG:teuthology.parallel:result is None 2024-09-12T07:57:34.719 INFO:teuthology.orchestra.run.smithi137.stdout:38654705669 2024-09-12T07:57:34.724 INFO:teuthology.orchestra.run.smithi137.stdout:38654705670 2024-09-12T07:57:34.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2024-09-12T07:57:34.740 DEBUG:teuthology.parallel:result is None 2024-09-12T07:57:34.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-09-12T07:57:34.742 DEBUG:teuthology.parallel:result is None 2024-09-12T07:57:34.743 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-12T07:57:34.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T07:57:35.067 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:35.068 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T07:57:35.085 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T07:57:33.864522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":86832,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283028688,"statfs":{"total":289910292480,"available":289821376512,"internally_reserved":0,"allocated":1613824,"data_stored":1302373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":87223631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.302196"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.593056+0000","last_change":"2024-09-12T07:57:31.593411+0000","last_active":"2024-09-12T07:57:31.593056+0000","last_peered":"2024-09-12T07:57:31.593056+0000","last_clean":"2024-09-12T07:57:31.593056+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T07:57:31.593056+0000","last_undegraded":"2024-09-12T07:57:31.593056+0000","last_fullsized":"2024-09-12T07:57:31.593056+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:47.434507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.592351+0000","last_change":"2024-09-12T07:57:31.592720+0000","last_active":"2024-09-12T07:57:31.592351+0000","last_peered":"2024-09-12T07:57:31.592351+0000","last_clean":"2024-09-12T07:57:31.592351+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T07:57:31.592351+0000","last_undegraded":"2024-09-12T07:57:31.592351+0000","last_fullsized":"2024-09-12T07:57:31.592351+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:40:49.569427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.592513+0000","last_change":"2024-09-12T07:57:31.592903+0000","last_active":"2024-09-12T07:57:31.592513+0000","last_peered":"2024-09-12T07:57:31.592513+0000","last_clean":"2024-09-12T07:57:31.592513+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T07:57:31.592513+0000","last_undegraded":"2024-09-12T07:57:31.592513+0000","last_fullsized":"2024-09-12T07:57:31.592513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:31.416374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.591845+0000","last_change":"2024-09-12T07:57:31.592249+0000","last_active":"2024-09-12T07:57:31.591845+0000","last_peered":"2024-09-12T07:57:31.591845+0000","last_clean":"2024-09-12T07:57:31.591845+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T07:57:31.591845+0000","last_undegraded":"2024-09-12T07:57:31.591845+0000","last_fullsized":"2024-09-12T07:57:31.591845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:43:47.133587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.586790+0000","last_change":"2024-09-12T07:57:31.586790+0000","last_active":"2024-09-12T07:57:31.586790+0000","last_peered":"2024-09-12T07:57:31.586790+0000","last_clean":"2024-09-12T07:57:31.586790+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T07:57:31.586790+0000","last_undegraded":"2024-09-12T07:57:31.586790+0000","last_fullsized":"2024-09-12T07:57:31.586790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:26:12.712604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.576874+0000","last_change":"2024-09-12T07:57:31.577407+0000","last_active":"2024-09-12T07:57:31.576874+0000","last_peered":"2024-09-12T07:57:31.576874+0000","last_clean":"2024-09-12T07:57:31.576874+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T07:57:31.576874+0000","last_undegraded":"2024-09-12T07:57:31.576874+0000","last_fullsized":"2024-09-12T07:57:31.576874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:26:14.289003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.576798+0000","last_change":"2024-09-12T07:57:31.577181+0000","last_active":"2024-09-12T07:57:31.576798+0000","last_peered":"2024-09-12T07:57:31.576798+0000","last_clean":"2024-09-12T07:57:31.576798+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T07:57:31.576798+0000","last_undegraded":"2024-09-12T07:57:31.576798+0000","last_fullsized":"2024-09-12T07:57:31.576798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:59:10.221338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.591758+0000","last_change":"2024-09-12T07:57:31.592415+0000","last_active":"2024-09-12T07:57:31.591758+0000","last_peered":"2024-09-12T07:57:31.591758+0000","last_clean":"2024-09-12T07:57:31.591758+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T07:57:31.591758+0000","last_undegraded":"2024-09-12T07:57:31.591758+0000","last_fullsized":"2024-09-12T07:57:31.591758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:25:45.764022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.591108+0000","last_change":"2024-09-12T07:57:26.399925+0000","last_active":"2024-09-12T07:57:31.591108+0000","last_peered":"2024-09-12T07:57:31.591108+0000","last_clean":"2024-09-12T07:57:31.591108+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T07:57:31.591108+0000","last_undegraded":"2024-09-12T07:57:31.591108+0000","last_fullsized":"2024-09-12T07:57:31.591108+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_clean_scrub_stamp":"2024-09-12T07:57:24.546601+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:46:35.118087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":40533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":737280,"data_stored":630920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":737280,"data_stored":630920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T07:57:35.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T07:57:35.374 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:35.374 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T07:57:35.391 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T07:57:33.864522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":86832,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283028688,"statfs":{"total":289910292480,"available":289821376512,"internally_reserved":0,"allocated":1613824,"data_stored":1302373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":87223631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.302196"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.593056+0000","last_change":"2024-09-12T07:57:31.593411+0000","last_active":"2024-09-12T07:57:31.593056+0000","last_peered":"2024-09-12T07:57:31.593056+0000","last_clean":"2024-09-12T07:57:31.593056+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T07:57:31.593056+0000","last_undegraded":"2024-09-12T07:57:31.593056+0000","last_fullsized":"2024-09-12T07:57:31.593056+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:47.434507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.592351+0000","last_change":"2024-09-12T07:57:31.592720+0000","last_active":"2024-09-12T07:57:31.592351+0000","last_peered":"2024-09-12T07:57:31.592351+0000","last_clean":"2024-09-12T07:57:31.592351+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T07:57:31.592351+0000","last_undegraded":"2024-09-12T07:57:31.592351+0000","last_fullsized":"2024-09-12T07:57:31.592351+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:40:49.569427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.592513+0000","last_change":"2024-09-12T07:57:31.592903+0000","last_active":"2024-09-12T07:57:31.592513+0000","last_peered":"2024-09-12T07:57:31.592513+0000","last_clean":"2024-09-12T07:57:31.592513+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T07:57:31.592513+0000","last_undegraded":"2024-09-12T07:57:31.592513+0000","last_fullsized":"2024-09-12T07:57:31.592513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:31.416374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.591845+0000","last_change":"2024-09-12T07:57:31.592249+0000","last_active":"2024-09-12T07:57:31.591845+0000","last_peered":"2024-09-12T07:57:31.591845+0000","last_clean":"2024-09-12T07:57:31.591845+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T07:57:31.591845+0000","last_undegraded":"2024-09-12T07:57:31.591845+0000","last_fullsized":"2024-09-12T07:57:31.591845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:43:47.133587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.586790+0000","last_change":"2024-09-12T07:57:31.586790+0000","last_active":"2024-09-12T07:57:31.586790+0000","last_peered":"2024-09-12T07:57:31.586790+0000","last_clean":"2024-09-12T07:57:31.586790+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T07:57:31.586790+0000","last_undegraded":"2024-09-12T07:57:31.586790+0000","last_fullsized":"2024-09-12T07:57:31.586790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:26:12.712604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.576874+0000","last_change":"2024-09-12T07:57:31.577407+0000","last_active":"2024-09-12T07:57:31.576874+0000","last_peered":"2024-09-12T07:57:31.576874+0000","last_clean":"2024-09-12T07:57:31.576874+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T07:57:31.576874+0000","last_undegraded":"2024-09-12T07:57:31.576874+0000","last_fullsized":"2024-09-12T07:57:31.576874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:26:14.289003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.576798+0000","last_change":"2024-09-12T07:57:31.577181+0000","last_active":"2024-09-12T07:57:31.576798+0000","last_peered":"2024-09-12T07:57:31.576798+0000","last_clean":"2024-09-12T07:57:31.576798+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T07:57:31.576798+0000","last_undegraded":"2024-09-12T07:57:31.576798+0000","last_fullsized":"2024-09-12T07:57:31.576798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:59:10.221338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.591758+0000","last_change":"2024-09-12T07:57:31.592415+0000","last_active":"2024-09-12T07:57:31.591758+0000","last_peered":"2024-09-12T07:57:31.591758+0000","last_clean":"2024-09-12T07:57:31.591758+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T07:57:31.591758+0000","last_undegraded":"2024-09-12T07:57:31.591758+0000","last_fullsized":"2024-09-12T07:57:31.591758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:25:45.764022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-12T07:57:31.591108+0000","last_change":"2024-09-12T07:57:26.399925+0000","last_active":"2024-09-12T07:57:31.591108+0000","last_peered":"2024-09-12T07:57:31.591108+0000","last_clean":"2024-09-12T07:57:31.591108+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T07:57:31.591108+0000","last_undegraded":"2024-09-12T07:57:31.591108+0000","last_fullsized":"2024-09-12T07:57:31.591108+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_clean_scrub_stamp":"2024-09-12T07:57:24.546601+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:46:35.118087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":40533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":737280,"data_stored":630920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":737280,"data_stored":630920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T07:57:35.392 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-12T07:57:35.392 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-12T07:57:35.392 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-12T07:57:35.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-12T07:57:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T07:57:35.700 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-12T07:57:35.719 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-12T07:57:35.719 INFO:teuthology.run_tasks:Running task exec... 2024-09-12T07:57:35.731 INFO:teuthology.task.exec:Executing custom commands... 2024-09-12T07:57:35.731 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T07:57:35.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2024-09-12T07:57:36.593 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'datapool' created 2024-09-12T07:57:36.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2024-09-12T07:57:38.919 INFO:teuthology.run_tasks:Running task install... 2024-09-12T07:57:38.928 DEBUG:teuthology.task.install:project ceph 2024-09-12T07:57:38.928 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}} 2024-09-12T07:57:38.928 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'} 2024-09-12T07:57:38.928 INFO:teuthology.task.install:Using flavor: default 2024-09-12T07:57:38.933 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-12T07:57:38.934 INFO:teuthology.task.install:extra packages: [] 2024-09-12T07:57:38.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-key list | grep Ceph 2024-09-12T07:57:38.996 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-12T07:57:39.022 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-12T07:57:39.022 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph.com (release key) 2024-09-12T07:57:39.023 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-12T07:57:39.023 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2024-09-12T07:57:39.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:57:39.185 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default/ 2024-09-12T07:57:39.185 INFO:teuthology.task.install.deb:Package version is 19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:57:39.307 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:57:39.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-12T07:57:39.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-12T07:57:39.545 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy InRelease 2024-09-12T07:57:39.566 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy Release 2024-09-12T07:57:39.589 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-12T07:57:39.623 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-12T07:57:39.623 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-12T07:57:39.698 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-12T07:57:39.783 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-12T07:57:41.017 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:57:41.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4883-gd61b0886-1jammy cephadm=19.3.0-4883-gd61b0886-1jammy ceph-mds=19.3.0-4883-gd61b0886-1jammy ceph-mgr=19.3.0-4883-gd61b0886-1jammy ceph-common=19.3.0-4883-gd61b0886-1jammy ceph-fuse=19.3.0-4883-gd61b0886-1jammy ceph-test=19.3.0-4883-gd61b0886-1jammy ceph-volume=19.3.0-4883-gd61b0886-1jammy radosgw=19.3.0-4883-gd61b0886-1jammy python3-rados=19.3.0-4883-gd61b0886-1jammy python3-rgw=19.3.0-4883-gd61b0886-1jammy python3-cephfs=19.3.0-4883-gd61b0886-1jammy python3-rbd=19.3.0-4883-gd61b0886-1jammy libcephfs2=19.3.0-4883-gd61b0886-1jammy libcephfs-dev=19.3.0-4883-gd61b0886-1jammy librados2=19.3.0-4883-gd61b0886-1jammy librbd1=19.3.0-4883-gd61b0886-1jammy rbd-fuse=19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:57:41.088 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:57:41.275 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-12T07:57:41.276 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-12T07:57:41.459 INFO:teuthology.orchestra.run.smithi137.stdout:ceph is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.459 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-common is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.459 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-fuse is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.459 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-mds is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-mgr is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-test is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-volume is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-dev is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs2 is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:librados2 is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:librbd1 is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:python3-cephfs is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:python3-rados is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:python3-rbd is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.460 INFO:teuthology.orchestra.run.smithi137.stdout:python3-rgw is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout:radosgw is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse is already the newest version (19.3.0-4883-gd61b0886-1jammy). 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-12T07:57:41.461 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-12T07:57:41.498 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 331 not upgraded. 2024-09-12T07:57:41.499 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-12T07:57:41.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2024-09-12T07:57:41.609 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-12T07:57:41.818 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-12T07:57:41.818 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-12T07:57:42.015 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-12T07:57:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2024-09-12T07:57:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2024-09-12T07:57:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2024-09-12T07:57:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2024-09-12T07:57:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: python2.7 python2.7-dev python2.7-minimal 2024-09-12T07:57:42.016 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-12T07:57:42.017 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-09-12T07:57:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemblk1 2024-09-12T07:57:42.018 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-09-12T07:57:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: gnuplot gfio python-scipy 2024-09-12T07:57:42.043 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-09-12T07:57:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: fio libpmemblk1 2024-09-12T07:57:42.226 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 2 newly installed, 0 to remove and 331 not upgraded. 2024-09-12T07:57:42.226 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 4,112 kB of archives. 2024-09-12T07:57:42.226 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 7,813 kB of additional disk space will be used. 2024-09-12T07:57:42.226 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2024-09-12T07:57:42.515 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2024-09-12T07:57:43.390 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 4,112 kB in 1s (4,224 kB/s) 2024-09-12T07:57:43.556 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libpmemblk1:amd64. 2024-09-12T07:57:43.661 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176253 files and directories currently installed.) 2024-09-12T07:57:43.667 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2024-09-12T07:57:43.755 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2024-09-12T07:57:44.042 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package fio. 2024-09-12T07:57:44.070 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2024-09-12T07:57:44.108 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking fio (3.28-1) ... 2024-09-12T07:57:44.677 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2024-09-12T07:57:44.855 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up fio (3.28-1) ... 2024-09-12T07:57:45.316 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-12T07:57:46.292 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-12T07:57:46.717 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-12T07:57:46.721 DEBUG:teuthology.parallel:result is None 2024-09-12T07:57:46.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:57:46.886 DEBUG:teuthology.orchestra.run.smithi137:> dpkg-query -W -f '${Version}' ceph 2024-09-12T07:57:46.904 INFO:teuthology.orchestra.run.smithi137.stdout:19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:57:46.904 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883-gd61b0886-1jammy 2024-09-12T07:57:46.905 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883-gd61b0886-1jammy is installed. 2024-09-12T07:57:46.906 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2024-09-12T07:57:46.906 INFO:teuthology.run_tasks:Running task workunit... 2024-09-12T07:57:46.920 INFO:tasks.workunit:Pulling workunits from ref d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:57:46.920 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-12T07:57:46.920 DEBUG:teuthology.orchestra.run.smithi137:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-12T07:57:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:57:46.954 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-12T07:57:46.954 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-12T07:57:47.001 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-12T07:57:47.001 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-12T07:57:47.050 INFO:tasks.workunit:timeout=3h 2024-09-12T07:57:47.050 INFO:tasks.workunit:cleanup=True 2024-09-12T07:57:47.050 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T07:57:47.099 INFO:tasks.workunit.client.0.smithi137.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-12T07:58:35.583 INFO:tasks.workunit.client.0.smithi137.stderr:Note: switching to 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'. 2024-09-12T07:58:35.583 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr:state without impacting any branches by switching back to a branch. 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr: git switch -c 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr:Or undo this operation with: 2024-09-12T07:58:35.584 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.585 INFO:tasks.workunit.client.0.smithi137.stderr: git switch - 2024-09-12T07:58:35.585 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.585 INFO:tasks.workunit.client.0.smithi137.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-12T07:58:35.585 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-09-12T07:58:35.585 INFO:tasks.workunit.client.0.smithi137.stderr:HEAD is now at d61b08865d4 Merge branch 'wip-nbalacha-ns-mirroring' of https://github.com/nbalacha/ceph into HEAD 2024-09-12T07:58:35.589 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-12T07:58:35.637 INFO:tasks.workunit.client.0.smithi137.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-12T07:58:35.640 INFO:tasks.workunit.client.0.smithi137.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-12T07:58:35.640 INFO:tasks.workunit.client.0.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-12T07:58:36.384 INFO:tasks.workunit.client.0.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-12T07:58:36.465 INFO:tasks.workunit.client.0.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-12T07:58:36.522 INFO:tasks.workunit.client.0.smithi137.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-12T07:58:36.524 INFO:tasks.workunit.client.0.smithi137.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-12T07:58:36.524 INFO:tasks.workunit.client.0.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-12T07:58:36.569 INFO:tasks.workunit.client.0.smithi137.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-12T07:58:36.572 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-12T07:58:36.572 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-12T07:58:36.621 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2024-09-12T07:58:36.623 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2024-09-12T07:58:36.623 DEBUG:teuthology.orchestra.run.smithi137:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2024-09-12T07:58:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ POOL=rbd 2024-09-12T07:58:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ IMAGE_PREFIX=image 2024-09-12T07:58:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ NUM_IMAGES=20 2024-09-12T07:58:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ RUN_TIME=3600 2024-09-12T07:58:36.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror pool enable rbd image 2024-09-12T07:58:36.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror pool peer add rbd dummy 2024-09-12T07:58:36.758 INFO:tasks.workunit.client.0.smithi137.stdout:79eec6a7-ef5b-4a71-8df0-b04be4dbf504 2024-09-12T07:58:36.761 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i = 1 )) 2024-09-12T07:58:36.762 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:36.762 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2024-09-12T07:58:36.809 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image1 snapshot 2024-09-12T07:58:36.953 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:36.964 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2024-09-12T07:58:37.025 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:37.025 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:37.025 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2024-09-12T07:58:37.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image2 snapshot 2024-09-12T07:58:37.970 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:37.984 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2024-09-12T07:58:38.047 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:38.047 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:38.047 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2024-09-12T07:58:38.096 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image3 snapshot 2024-09-12T07:58:38.972 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:38.988 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2024-09-12T07:58:39.050 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:39.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:39.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2024-09-12T07:58:39.106 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image4 snapshot 2024-09-12T07:58:39.983 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:39.995 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2024-09-12T07:58:40.060 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:40.060 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:40.060 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2024-09-12T07:58:40.112 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image5 snapshot 2024-09-12T07:58:40.993 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:41.005 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2024-09-12T07:58:41.066 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:41.066 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:41.066 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2024-09-12T07:58:41.116 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image6 snapshot 2024-09-12T07:58:42.001 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:42.014 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2024-09-12T07:58:42.076 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:42.076 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:42.076 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2024-09-12T07:58:42.126 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image7 snapshot 2024-09-12T07:58:43.005 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:43.017 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2024-09-12T07:58:43.079 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:43.079 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:43.079 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2024-09-12T07:58:43.133 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image8 snapshot 2024-09-12T07:58:44.007 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:44.021 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2024-09-12T07:58:44.082 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:44.082 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:44.082 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2024-09-12T07:58:44.132 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image9 snapshot 2024-09-12T07:58:45.021 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:45.031 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2024-09-12T07:58:45.092 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:45.092 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:45.092 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2024-09-12T07:58:45.141 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image10 snapshot 2024-09-12T07:58:46.024 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:46.197 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2024-09-12T07:58:46.197 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:46.197 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:46.198 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2024-09-12T07:58:46.198 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image11 snapshot 2024-09-12T07:58:47.035 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:47.050 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2024-09-12T07:58:47.112 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:47.112 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:47.112 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2024-09-12T07:58:47.162 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image12 snapshot 2024-09-12T07:58:48.035 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:48.047 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2024-09-12T07:58:48.107 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:48.107 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:48.108 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2024-09-12T07:58:48.156 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image13 snapshot 2024-09-12T07:58:49.043 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:49.053 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2024-09-12T07:58:49.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:49.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:49.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2024-09-12T07:58:49.169 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image14 snapshot 2024-09-12T07:58:50.047 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:50.059 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2024-09-12T07:58:50.120 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:50.121 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:50.121 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2024-09-12T07:58:50.169 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image15 snapshot 2024-09-12T07:58:51.066 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:51.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2024-09-12T07:58:51.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:51.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:51.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2024-09-12T07:58:51.198 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image16 snapshot 2024-09-12T07:58:52.071 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:52.085 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2024-09-12T07:58:52.147 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:52.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:52.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2024-09-12T07:58:52.196 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image17 snapshot 2024-09-12T07:58:53.074 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:53.088 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2024-09-12T07:58:53.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:53.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:53.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2024-09-12T07:58:53.196 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image18 snapshot 2024-09-12T07:58:54.084 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:54.097 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2024-09-12T07:58:54.159 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:54.159 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:54.159 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2024-09-12T07:58:54.211 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image19 snapshot 2024-09-12T07:58:55.086 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:55.098 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2024-09-12T07:58:55.159 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:55.160 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:55.160 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2024-09-12T07:58:55.213 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror image enable rbd/image20 snapshot 2024-09-12T07:58:56.095 INFO:tasks.workunit.client.0.smithi137.stdout:Mirroring enabled 2024-09-12T07:58:56.109 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2024-09-12T07:58:56.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i = 1 )) 2024-09-12T07:58:56.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.171 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image1 2024-09-12T07:58:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd0 2024-09-12T07:58:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image2 2024-09-12T07:58:56.367 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd1 2024-09-12T07:58:56.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.368 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image3 2024-09-12T07:58:56.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd2 2024-09-12T07:58:56.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.440 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image4 2024-09-12T07:58:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd3 2024-09-12T07:58:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image5 2024-09-12T07:58:56.587 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd4 2024-09-12T07:58:56.587 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.587 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.588 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image6 2024-09-12T07:58:56.588 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd5 2024-09-12T07:58:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image7 2024-09-12T07:58:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd6 2024-09-12T07:58:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.789 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image8 2024-09-12T07:58:56.898 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd7 2024-09-12T07:58:56.898 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:56.898 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:56.898 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:56.898 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image9 2024-09-12T07:58:57.031 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd8 2024-09-12T07:58:57.031 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.032 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.032 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.032 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image10 2024-09-12T07:58:57.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd9 2024-09-12T07:58:57.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.148 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.156 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image11 2024-09-12T07:58:57.282 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd10 2024-09-12T07:58:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image12 2024-09-12T07:58:57.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd11 2024-09-12T07:58:57.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.434 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.434 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.434 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image13 2024-09-12T07:58:57.567 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd12 2024-09-12T07:58:57.567 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.567 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.568 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image14 2024-09-12T07:58:57.568 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd13 2024-09-12T07:58:57.693 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.693 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.693 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.693 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image15 2024-09-12T07:58:57.829 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd14 2024-09-12T07:58:57.829 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.829 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.829 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.829 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image16 2024-09-12T07:58:57.953 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd15 2024-09-12T07:58:57.953 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:57.953 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:57.953 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:57.953 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image17 2024-09-12T07:58:58.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd16 2024-09-12T07:58:58.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:58.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:58.081 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image18 2024-09-12T07:58:58.088 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:58.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd17 2024-09-12T07:58:58.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:58.260 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:58.260 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:58.260 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image19 2024-09-12T07:58:58.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd18 2024-09-12T07:58:58.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:58.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:58.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:58.449 INFO:tasks.workunit.client.0.smithi137.stderr:++ sudo rbd device map rbd/image20 2024-09-12T07:58:58.587 INFO:tasks.workunit.client.0.smithi137.stderr:+ DEVS[$i]=/dev/rbd19 2024-09-12T07:58:58.588 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T07:58:58.588 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T07:58:58.588 INFO:tasks.workunit.client.0.smithi137.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2024-09-12T07:58:58.588 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726127938 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ END_TIME=1726131538 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR= 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR= 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n '' ]] 2024-09-12T07:58:58.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T07:59:03.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T07:59:03.959 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T07:59:03.959 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T07:59:03.959 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T07:59:03.959 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T07:59:03.959 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T07:59:03.960 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T07:59:03.960 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T07:59:03.960 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T07:59:03.960 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T07:59:03.960 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T07:59:03.961 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T07:59:03.961 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T07:59:03.956+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T07:59:08.592 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T07:59:08.592 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T07:59:08.595 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T07:59:08.596 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T07:59:09.517 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1667139831 2024-09-12T07:59:09.517 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T07:59:09.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726127949 2024-09-12T07:59:09.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T07:59:09.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1667139831 ]] 2024-09-12T07:59:09.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1667139831 != '' ]] 2024-09-12T07:59:09.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1667139831 2024-09-12T07:59:11.965 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1667139831 until 2024-09-12T08:59:11.244604+0000 (3600 sec) 2024-09-12T07:59:11.997 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T07:59:11.997 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1667139831 2024-09-12T07:59:12.783 INFO:tasks.workunit.client.0.smithi137.stderr:listed 1 entries 2024-09-12T07:59:12.816 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1667139831 2024-09-12T07:59:12.816 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T07:59:21.046 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T07:59:21.044+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T07:59:21.046 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T07:59:21.044+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T07:59:22.819 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T07:59:22.819 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T07:59:22.819 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T07:59:22.826 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T07:59:23.769 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1667139831 2024-09-12T07:59:23.769 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T07:59:23.769 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726127963 2024-09-12T07:59:23.769 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T07:59:23.770 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1667139831 ]] 2024-09-12T07:59:23.770 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1667139831 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\7\1\3\9\8\3\1 ]] 2024-09-12T07:59:23.770 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T07:59:33.776 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T07:59:33.777 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T07:59:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T07:59:33.778 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T07:59:34.868 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T07:59:34.868 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T07:59:34.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726127974 2024-09-12T07:59:34.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T07:59:34.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2562768379 ]] 2024-09-12T07:59:34.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2562768379 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\7\1\3\9\8\3\1 ]] 2024-09-12T07:59:34.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2562768379 2024-09-12T07:59:36.951 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2562768379 until 2024-09-12T08:59:36.176533+0000 (3600 sec) 2024-09-12T07:59:36.991 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T07:59:36.992 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2562768379 2024-09-12T07:59:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:listed 2 entries 2024-09-12T07:59:37.868 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T07:59:37.868 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T07:59:47.873 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T07:59:47.873 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T07:59:47.873 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T07:59:47.874 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T07:59:48.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T07:59:48.884 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T07:59:48.884 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726127988 2024-09-12T07:59:48.884 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T07:59:48.884 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2562768379 ]] 2024-09-12T07:59:48.884 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2562768379 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\6\2\7\6\8\3\7\9 ]] 2024-09-12T07:59:48.885 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T07:59:58.885 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T07:59:58.886 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T07:59:58.890 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T07:59:58.895 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T07:59:59.819 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T07:59:59.819 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T07:59:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726127999 2024-09-12T07:59:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T07:59:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2562768379 ]] 2024-09-12T07:59:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2562768379 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\6\2\7\6\8\3\7\9 ]] 2024-09-12T07:59:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:00:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.016+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e899b080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.032+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.036+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.036+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.040+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.044+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.044+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.044+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944e880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.048+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.048+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.052+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.052+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e90ee300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.056+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.056+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.060+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.060+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.064+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.064+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.072+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.072+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.076+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.076+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944e880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.076+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.076+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e899a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.084+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.084+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.088+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:00:00.088+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:00:09.822 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:00:09.822 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:00:09.823 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:00:09.823 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:00:10.879 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T08:00:10.879 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:00:10.880 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128010 2024-09-12T08:00:10.880 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:00:10.880 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2562768379 ]] 2024-09-12T08:00:10.880 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2562768379 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\6\2\7\6\8\3\7\9 ]] 2024-09-12T08:00:10.880 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:00:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:00:20.882 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:00:20.887 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:00:20.888 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:00:21.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T08:00:21.911 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:00:21.911 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128021 2024-09-12T08:00:21.911 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:00:21.911 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2562768379 ]] 2024-09-12T08:00:21.912 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2562768379 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\6\2\7\6\8\3\7\9 ]] 2024-09-12T08:00:21.912 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:00:31.914 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:00:31.919 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:00:31.926 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:00:31.933 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:00:32.882 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2562768379 2024-09-12T08:00:32.891 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:00:32.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128032 2024-09-12T08:00:32.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:00:32.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2562768379 ]] 2024-09-12T08:00:32.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2562768379 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\6\2\7\6\8\3\7\9 ]] 2024-09-12T08:00:32.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:00:42.894 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:00:42.894 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:00:42.894 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:00:42.895 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:00:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:00:43.872 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:00:43.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128043 2024-09-12T08:00:43.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:00:43.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2035321561 ]] 2024-09-12T08:00:43.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2035321561 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\6\2\7\6\8\3\7\9 ]] 2024-09-12T08:00:43.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2035321561 2024-09-12T08:00:45.992 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2035321561 until 2024-09-12T09:00:45.267612+0000 (3600 sec) 2024-09-12T08:00:46.035 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2035321561 2024-09-12T08:00:46.038 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:00:46.791 INFO:tasks.workunit.client.0.smithi137.stderr:listed 3 entries 2024-09-12T08:00:46.822 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:00:46.823 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:00:56.839 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:00:56.846 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:00:56.848 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:00:56.849 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:00:57.947 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:00:57.948 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:00:57.948 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128057 2024-09-12T08:00:57.948 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:00:57.948 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2035321561 ]] 2024-09-12T08:00:57.948 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2035321561 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\3\5\3\2\1\5\6\1 ]] 2024-09-12T08:00:57.949 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:01:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.011+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.011+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.011+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.011+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.015+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.015+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.023+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.023+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9635980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.023+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.023+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9635800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.027+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.027+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.031+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.031+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.031+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.031+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.035+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.035+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e944f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.035+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.039+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.039+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.039+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.043+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.043+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9635700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.043+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.047+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9635900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.047+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.047+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.051+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.051+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.055+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9635b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9635780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:01:07.950 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:01:07.953 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:01:07.957 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:01:07.976 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:01:08.914 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:01:08.914 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:01:08.915 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128068 2024-09-12T08:01:08.915 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:01:08.915 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2035321561 ]] 2024-09-12T08:01:08.915 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2035321561 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\3\5\3\2\1\5\6\1 ]] 2024-09-12T08:01:08.915 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:01:14.573 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:01:14.571+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:01:14.573 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:01:14.571+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:01:18.917 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:01:18.918 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:01:18.918 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:01:18.923 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:01:19.899 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:01:19.899 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:01:19.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128079 2024-09-12T08:01:19.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:01:19.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2035321561 ]] 2024-09-12T08:01:19.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2035321561 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\3\5\3\2\1\5\6\1 ]] 2024-09-12T08:01:19.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:01:29.902 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:01:29.903 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:01:29.906 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:01:29.912 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:01:30.966 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:01:30.966 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:01:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128090 2024-09-12T08:01:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:01:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2035321561 ]] 2024-09-12T08:01:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2035321561 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\3\5\3\2\1\5\6\1 ]] 2024-09-12T08:01:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:01:40.501 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:01:40.499+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:01:40.501 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:01:40.499+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:01:40.969 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:01:40.969 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:01:40.969 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:01:40.976 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:01:42.066 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2035321561 2024-09-12T08:01:42.066 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:01:42.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128102 2024-09-12T08:01:42.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:01:42.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2035321561 ]] 2024-09-12T08:01:42.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2035321561 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\3\5\3\2\1\5\6\1 ]] 2024-09-12T08:01:42.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:01:49.751 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:01:49.751+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:01:49.751 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:01:49.751+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:01:52.068 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:01:52.069 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:01:52.069 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:01:52.081 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:01:53.042 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:01:53.043 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:01:53.043 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128113 2024-09-12T08:01:53.044 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:01:53.044 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3433447522 ]] 2024-09-12T08:01:53.044 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3433447522 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\3\5\3\2\1\5\6\1 ]] 2024-09-12T08:01:53.044 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3433447522 2024-09-12T08:01:54.974 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3433447522 until 2024-09-12T09:01:54.228148+0000 (3600 sec) 2024-09-12T08:01:55.031 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:01:55.039 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3433447522 2024-09-12T08:01:56.000 INFO:tasks.workunit.client.0.smithi137.stderr:listed 4 entries 2024-09-12T08:01:56.026 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:01:56.026 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:02:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:59.999+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:01:59.999+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95a9e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e90eed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9635b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e90efe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.015+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.015+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95a9e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95a9880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9635800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9635b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.051+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.051+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e90ef180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.091+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:02:00.091+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:02:04.736 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:02:04.736 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:02:04.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:02:04.735+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:02:06.038 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:02:06.038 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:02:06.056 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:02:06.061 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:02:07.045 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:02:07.045 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:02:07.046 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128127 2024-09-12T08:02:07.046 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:02:07.046 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3433447522 ]] 2024-09-12T08:02:07.046 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3433447522 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\3\3\4\4\7\5\2\2 ]] 2024-09-12T08:02:07.046 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:02:17.061 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:02:17.062 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:02:17.062 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:02:17.072 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:02:18.073 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:02:18.073 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:02:18.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128138 2024-09-12T08:02:18.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:02:18.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3433447522 ]] 2024-09-12T08:02:18.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3433447522 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\3\3\4\4\7\5\2\2 ]] 2024-09-12T08:02:18.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:02:28.090 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:02:28.090 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:02:28.090 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:02:28.095 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:02:29.316 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:02:29.316 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:02:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128149 2024-09-12T08:02:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:02:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3433447522 ]] 2024-09-12T08:02:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3433447522 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\3\3\4\4\7\5\2\2 ]] 2024-09-12T08:02:29.317 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:02:39.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:02:39.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:02:39.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:02:39.340 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:02:40.323 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:02:40.323 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:02:40.324 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128160 2024-09-12T08:02:40.325 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:02:40.325 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3433447522 ]] 2024-09-12T08:02:40.325 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3433447522 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\3\3\4\4\7\5\2\2 ]] 2024-09-12T08:02:40.325 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:02:50.326 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:02:50.346 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:02:50.347 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:02:50.359 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:02:51.454 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3433447522 2024-09-12T08:02:51.454 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:02:51.455 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128171 2024-09-12T08:02:51.455 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:02:51.455 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3433447522 ]] 2024-09-12T08:02:51.455 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3433447522 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\3\3\4\4\7\5\2\2 ]] 2024-09-12T08:02:51.455 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:03:01.457 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:03:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:03:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:03:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:03:02.443 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:03:02.444 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:03:02.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128182 2024-09-12T08:03:02.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:03:02.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2340344547 ]] 2024-09-12T08:03:02.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2340344547 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\3\3\4\4\7\5\2\2 ]] 2024-09-12T08:03:02.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2340344547 2024-09-12T08:03:04.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.106+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e9962900 image watch failed: 94042229339776, (107) Transport endpoint is not connected 2024-09-12T08:03:04.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.106+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9962900 handle_error: handle=94042229339776: (107) Transport endpoint is not connected 2024-09-12T08:03:04.115 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::managed_lock::BreakRequest: 0x5587e9b32b40 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:03:04.115 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::managed_lock::AcquireRequest: 0x5587e964ef00 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2024-09-12T08:03:04.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::ManagedLock: 0x5587e88e2b78 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:03:04.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e965d540 handle_unwatch client blocklisted 2024-09-12T08:03:04.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9962900 handle_rewatch: client blocklisted 2024-09-12T08:03:04.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904f860 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:03:04.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:03:04.118+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e94668c0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:03:05.018 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2340344547 until 2024-09-12T09:03:04.262948+0000 (3600 sec) 2024-09-12T08:03:05.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2340344547 2024-09-12T08:03:05.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:03:05.784 INFO:tasks.workunit.client.0.smithi137.stderr:listed 5 entries 2024-09-12T08:03:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:03:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:03:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:03:15.814 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:03:15.815 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:03:15.830 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:03:16.822 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:03:16.823 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:03:16.832 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128196 2024-09-12T08:03:16.832 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:03:16.833 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2340344547 ]] 2024-09-12T08:03:16.833 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2340344547 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\3\4\0\3\4\4\5\4\7 ]] 2024-09-12T08:03:16.833 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:03:26.834 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:03:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:03:26.850 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:03:26.853 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:03:27.754 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:03:27.755 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:03:27.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128207 2024-09-12T08:03:27.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:03:27.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2340344547 ]] 2024-09-12T08:03:27.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2340344547 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\3\4\0\3\4\4\5\4\7 ]] 2024-09-12T08:03:27.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:03:37.758 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:03:37.763 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:03:37.769 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:03:37.774 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:03:38.748 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:03:38.754 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:03:38.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128218 2024-09-12T08:03:38.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:03:38.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2340344547 ]] 2024-09-12T08:03:38.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2340344547 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\3\4\0\3\4\4\5\4\7 ]] 2024-09-12T08:03:38.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:03:48.771 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:03:48.772 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:03:48.772 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:03:48.772 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:03:49.694 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:03:49.694 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:03:49.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128229 2024-09-12T08:03:49.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:03:49.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2340344547 ]] 2024-09-12T08:03:49.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2340344547 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\3\4\0\3\4\4\5\4\7 ]] 2024-09-12T08:03:49.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:03:59.698 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:03:59.698 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:03:59.698 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:03:59.703 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:04:00.700 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2340344547 2024-09-12T08:04:00.701 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:04:00.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128240 2024-09-12T08:04:00.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:04:00.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2340344547 ]] 2024-09-12T08:04:00.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2340344547 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\3\4\0\3\4\4\5\4\7 ]] 2024-09-12T08:04:00.702 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:04:10.703 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:04:10.705 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:04:10.706 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:04:10.714 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:04:11.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:04:11.678 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:04:11.679 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128251 2024-09-12T08:04:11.679 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:04:11.679 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/708566858 ]] 2024-09-12T08:04:11.679 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/708566858 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\3\4\0\3\4\4\5\4\7 ]] 2024-09-12T08:04:11.679 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/708566858 2024-09-12T08:04:14.058 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/708566858 until 2024-09-12T09:04:13.274062+0000 (3600 sec) 2024-09-12T08:04:14.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/708566858 2024-09-12T08:04:14.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:04:14.877 INFO:tasks.workunit.client.0.smithi137.stderr:listed 6 entries 2024-09-12T08:04:14.906 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:04:14.906 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:04:24.924 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:04:24.929 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:04:24.936 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:04:24.938 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:04:25.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:04:25.893 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:04:25.893 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128265 2024-09-12T08:04:25.893 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:04:25.893 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/708566858 ]] 2024-09-12T08:04:25.893 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/708566858 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\0\8\5\6\6\8\5\8 ]] 2024-09-12T08:04:25.893 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:04:28.466 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.465+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:04:28.467 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:04:28.467 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:04:28.467 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:04:28.467 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:04:28.468 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:04:28.468 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:04:28.468 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:04:28.468 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:04:28.468 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:04:28.469+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:04:35.904 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:04:35.905 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:04:35.910 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:04:35.913 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:04:36.860 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:04:36.860 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:04:36.861 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128276 2024-09-12T08:04:36.861 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:04:36.861 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/708566858 ]] 2024-09-12T08:04:36.861 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/708566858 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\0\8\5\6\6\8\5\8 ]] 2024-09-12T08:04:36.861 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:04:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:04:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:04:46.884 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:04:46.892 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:04:47.826 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:04:47.827 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:04:47.827 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128287 2024-09-12T08:04:47.827 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:04:47.827 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/708566858 ]] 2024-09-12T08:04:47.827 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/708566858 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\0\8\5\6\6\8\5\8 ]] 2024-09-12T08:04:47.828 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:04:57.829 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:04:57.830 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:04:57.837 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:04:57.838 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:04:58.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:04:58.755 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:04:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128298 2024-09-12T08:04:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:04:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/708566858 ]] 2024-09-12T08:04:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/708566858 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\0\8\5\6\6\8\5\8 ]] 2024-09-12T08:04:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:05:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.000+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.000+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98ce100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98ced00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98cf700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98cf500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.016+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98cf300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.020+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.020+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.024+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.024+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.032+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992d600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.032+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.036+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.036+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.040+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.040+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98cf100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.044+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.044+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.048+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.048+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.048+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.048+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.052+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.052+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.056+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.056+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9942080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.060+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.060+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9942280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.064+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.064+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9942480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.068+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:05:00.068+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9942680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:05:08.762 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:05:08.764 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:05:08.766 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:05:08.766 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:05:09.764 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/708566858 2024-09-12T08:05:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:05:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128309 2024-09-12T08:05:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:05:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/708566858 ]] 2024-09-12T08:05:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/708566858 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\0\8\5\6\6\8\5\8 ]] 2024-09-12T08:05:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:05:19.767 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:05:19.767 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:05:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:05:19.787 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:05:20.809 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:05:20.809 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:05:20.810 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128320 2024-09-12T08:05:20.810 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:05:20.810 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2085061616 ]] 2024-09-12T08:05:20.810 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2085061616 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\0\8\5\6\6\8\5\8 ]] 2024-09-12T08:05:20.810 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2085061616 2024-09-12T08:05:23.020 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2085061616 until 2024-09-12T09:05:22.218923+0000 (3600 sec) 2024-09-12T08:05:23.063 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:05:23.063 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2085061616 2024-09-12T08:05:23.771 INFO:tasks.workunit.client.0.smithi137.stderr:listed 7 entries 2024-09-12T08:05:23.796 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:05:23.796 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:05:32.159 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:05:32.160+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:05:32.159 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:05:32.160+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:05:33.799 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:05:33.799 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:05:33.801 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:05:33.801 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:05:34.891 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:05:34.891 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:05:34.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128334 2024-09-12T08:05:34.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:05:34.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2085061616 ]] 2024-09-12T08:05:34.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2085061616 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\8\5\0\6\1\6\1\6 ]] 2024-09-12T08:05:34.892 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:05:44.904 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:05:44.909 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:05:44.913 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:05:44.923 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:05:45.849 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:05:45.849 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:05:45.850 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128345 2024-09-12T08:05:45.850 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:05:45.850 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2085061616 ]] 2024-09-12T08:05:45.850 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2085061616 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\8\5\0\6\1\6\1\6 ]] 2024-09-12T08:05:45.850 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:05:55.852 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:05:55.852 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:05:55.859 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:05:55.879 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:05:56.801 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:05:56.802 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:05:56.802 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128356 2024-09-12T08:05:56.802 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:05:56.803 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2085061616 ]] 2024-09-12T08:05:56.803 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2085061616 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\8\5\0\6\1\6\1\6 ]] 2024-09-12T08:05:56.803 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:06:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.000+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.000+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e95a9700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.004+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.004+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9635e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.008+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.008+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9942e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.012+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.012+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9942180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.016+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.016+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.028+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.028+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.032+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.032+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.036+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.036+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.044+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.044+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.044+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.044+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.048+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.048+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.048+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.052+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.052+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.052+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.056+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.056+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.060+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.060+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9972600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.060+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.060+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.068+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.068+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9972a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.072+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.072+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9972c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.080+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.080+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9972e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.080+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:00.080+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9973000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:06:05.724 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:06:05.724 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:06:05.724 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:06:05.724 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:06:05.725 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:05.723+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:06:06.805 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:06:06.809 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:06:06.810 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:06:06.810 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:06:07.868 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:06:07.868 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:06:07.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128367 2024-09-12T08:06:07.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:06:07.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2085061616 ]] 2024-09-12T08:06:07.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2085061616 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\8\5\0\6\1\6\1\6 ]] 2024-09-12T08:06:07.869 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:06:13.176 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:13.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:06:13.176 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:13.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:06:13.176 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:13.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:06:13.176 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:13.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:06:17.871 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:06:17.871 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:06:17.871 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:06:17.901 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:06:18.938 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2085061616 2024-09-12T08:06:18.944 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:06:18.946 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128378 2024-09-12T08:06:18.947 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:06:18.947 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2085061616 ]] 2024-09-12T08:06:18.947 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2085061616 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\8\5\0\6\1\6\1\6 ]] 2024-09-12T08:06:18.947 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:06:28.960 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:06:28.960 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:06:28.962 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:06:28.974 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:06:30.022 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:06:30.022 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:06:30.023 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128390 2024-09-12T08:06:30.023 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:06:30.023 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:06:30.023 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\8\5\0\6\1\6\1\6 ]] 2024-09-12T08:06:30.023 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2900346027 2024-09-12T08:06:32.038 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2900346027 until 2024-09-12T09:06:31.351060+0000 (3600 sec) 2024-09-12T08:06:32.077 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:06:32.085 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2900346027 2024-09-12T08:06:32.832 INFO:tasks.workunit.client.0.smithi137.stderr:listed 8 entries 2024-09-12T08:06:32.873 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:06:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:06:42.879 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:06:42.879 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:06:42.883 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:06:42.895 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:06:43.857 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:06:43.857 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:06:43.858 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128403 2024-09-12T08:06:43.858 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:06:43.858 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:06:43.858 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:06:43.858 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:06:53.870 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:06:53.880 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:06:53.881 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:06:53.883 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:06:54.822 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:06:54.822 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:06:54.823 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128414 2024-09-12T08:06:54.823 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:06:54.823 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:06:54.823 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:06:54.823 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:06:55.351 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:06:55.351 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:06:55.352 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:06:55.352 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:06:55.352 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:06:55.353 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:06:55.353 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:06:55.353 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:06:55.353 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:06:55.353 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:06:55.353 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:06:55.351+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:07:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:59.999+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:06:59.999+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9634b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.003+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.003+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e89c7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.003+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.003+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e944f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.007+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.007+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e886c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.011+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.011+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e944e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.015+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.015+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e98cee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.027+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.027+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9720880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.031+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.031+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e98cea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.035+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.035+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.039+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.039+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.043+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.043+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.055+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.055+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e98cf780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.059+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.059+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e965f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.071+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.071+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.071+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.071+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.075+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.075+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e90ee900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.079+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.079+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e90ef900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.083+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.083+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e965e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.087+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.087+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e95a9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.095+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:00.095+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e965f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:07:04.825 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:07:04.825 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:07:04.825 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:07:04.829 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:07:05.760 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:07:05.761 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:07:05.762 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128425 2024-09-12T08:07:05.762 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:07:05.762 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:07:05.763 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:07:05.763 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:07:15.764 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:07:15.764 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:07:15.764 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:07:15.764 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:07:16.657 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:07:16.658 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:07:16.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128436 2024-09-12T08:07:16.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:07:16.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:07:16.659 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:07:16.659 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:07:26.660 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:07:26.660 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:07:26.669 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:07:26.672 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:07:27.514 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:07:27.514 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:07:27.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128447 2024-09-12T08:07:27.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:07:27.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:07:27.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:07:27.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:07:37.517 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:07:37.518 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:07:37.520 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:07:37.523 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:07:38.580 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2900346027 2024-09-12T08:07:38.581 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:07:38.581 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128458 2024-09-12T08:07:38.581 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:07:38.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2900346027 ]] 2024-09-12T08:07:38.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2900346027 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:07:38.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:07:48.593 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:07:48.602 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:07:48.607 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:07:48.608 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:07:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:07:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:07:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128469 2024-09-12T08:07:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:07:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1041426508 ]] 2024-09-12T08:07:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1041426508 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\0\0\3\4\6\0\2\7 ]] 2024-09-12T08:07:49.612 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1041426508 2024-09-12T08:07:51.350 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1041426508 until 2024-09-12T09:07:50.588010+0000 (3600 sec) 2024-09-12T08:07:51.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:07:51.409 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1041426508 2024-09-12T08:07:52.201 INFO:tasks.workunit.client.0.smithi137.stderr:listed 9 entries 2024-09-12T08:07:52.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:07:52.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:08:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:59.998+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:07:59.998+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9634180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.002+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.002+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e90efc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.002+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.002+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e965f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.014+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.014+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e90ef680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.014+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.014+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e992c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.018+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.018+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9942400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.022+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.022+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.030+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.030+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.034+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.034+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.034+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.034+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9973180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.038+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.038+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.038+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.038+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9943f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.042+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.042+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.046+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.046+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.050+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.050+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.054+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.054+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.066+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.066+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9972c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.070+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.070+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9973480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.074+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.074+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9973e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.078+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:08:00.078+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9973d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:08:02.232 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:08:02.232 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:08:02.232 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:08:02.236 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:08:03.228 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:08:03.228 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:08:03.229 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128483 2024-09-12T08:08:03.229 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:08:03.229 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1041426508 ]] 2024-09-12T08:08:03.229 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1041426508 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\4\1\4\2\6\5\0\8 ]] 2024-09-12T08:08:03.229 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:08:13.231 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:08:13.231 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:08:13.234 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:08:13.234 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:08:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:08:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:08:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128494 2024-09-12T08:08:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:08:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1041426508 ]] 2024-09-12T08:08:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1041426508 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\4\1\4\2\6\5\0\8 ]] 2024-09-12T08:08:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:08:24.214 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:08:24.214 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:08:24.215 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:08:24.224 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:08:25.267 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:08:25.267 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:08:25.268 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128505 2024-09-12T08:08:25.268 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:08:25.268 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1041426508 ]] 2024-09-12T08:08:25.268 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1041426508 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\4\1\4\2\6\5\0\8 ]] 2024-09-12T08:08:25.268 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:08:35.277 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:08:35.278 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:08:35.283 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:08:35.283 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128516 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1041426508 ]] 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1041426508 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\4\1\4\2\6\5\0\8 ]] 2024-09-12T08:08:36.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:08:46.030 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:08:46.031 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:08:46.032 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:08:46.032 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:08:46.032 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:08:46.033 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:08:46.033 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:08:46.033 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:08:46.033 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:08:46.033 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:08:46.033 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:08:46.034+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:08:46.296 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:08:46.297 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:08:46.298 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:08:46.313 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:08:47.322 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1041426508 2024-09-12T08:08:47.331 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:08:47.332 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128527 2024-09-12T08:08:47.332 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:08:47.332 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1041426508 ]] 2024-09-12T08:08:47.332 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1041426508 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\4\1\4\2\6\5\0\8 ]] 2024-09-12T08:08:47.332 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:08:57.333 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:08:57.334 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:08:57.337 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:08:57.340 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:08:58.293 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:08:58.293 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:08:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128538 2024-09-12T08:08:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:08:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1133434747 ]] 2024-09-12T08:08:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1133434747 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\4\1\4\2\6\5\0\8 ]] 2024-09-12T08:08:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1133434747 2024-09-12T08:09:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.006+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.006+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e992c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.006+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.006+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.014+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.014+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.014+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.014+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.018+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.018+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9721480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.030+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.030+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9943980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.038+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.038+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e910cc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.046+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.046+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9942f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.062+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.062+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e95a9400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.066+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.066+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e95a8780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.070+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.070+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9635600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.067 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1133434747 until 2024-09-12T09:08:59.289095+0000 (3600 sec) 2024-09-12T08:09:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.074+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.074+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9634500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.078+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.078+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9973400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.082+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.082+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9972700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.086+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.090+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e95a9b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.090+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.090+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9973500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.092 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.094+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.092 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.094+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9973800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.098+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.098+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9973a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.102+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.102+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.102+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9973c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:09:00.102+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b90100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:09:00.116 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:09:00.116 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1133434747 2024-09-12T08:09:00.853 INFO:tasks.workunit.client.0.smithi137.stderr:listed 10 entries 2024-09-12T08:09:00.882 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:09:00.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:09:10.890 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:09:10.893 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:09:10.898 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:09:10.903 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:09:11.776 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:09:11.776 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:09:11.779 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128551 2024-09-12T08:09:11.779 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:09:11.779 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1133434747 ]] 2024-09-12T08:09:11.780 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1133434747 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\3\4\3\4\7\4\7 ]] 2024-09-12T08:09:11.780 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:09:21.785 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:09:21.787 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:09:21.806 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:09:21.809 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:09:22.783 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:09:22.783 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:09:22.783 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128562 2024-09-12T08:09:22.784 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:09:22.784 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1133434747 ]] 2024-09-12T08:09:22.784 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1133434747 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\3\4\3\4\7\4\7 ]] 2024-09-12T08:09:22.784 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:09:32.792 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:09:32.792 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:09:32.800 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:09:32.801 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:09:33.969 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:09:33.969 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:09:33.970 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128573 2024-09-12T08:09:33.970 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:09:33.970 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1133434747 ]] 2024-09-12T08:09:33.970 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1133434747 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\3\4\3\4\7\4\7 ]] 2024-09-12T08:09:33.970 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:09:43.980 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:09:43.980 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:09:43.980 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:09:43.985 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:09:44.960 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:09:44.960 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:09:44.960 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128584 2024-09-12T08:09:44.960 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:09:44.960 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1133434747 ]] 2024-09-12T08:09:44.961 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1133434747 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\3\4\3\4\7\4\7 ]] 2024-09-12T08:09:44.961 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:09:54.967 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:09:54.969 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:09:54.984 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:09:54.988 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:09:56.006 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1133434747 2024-09-12T08:09:56.006 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:09:56.007 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128596 2024-09-12T08:09:56.007 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:09:56.007 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1133434747 ]] 2024-09-12T08:09:56.007 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1133434747 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\3\4\3\4\7\4\7 ]] 2024-09-12T08:09:56.007 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:10:06.009 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:10:06.009 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:10:06.011 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:10:06.016 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:10:07.010 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:10:07.010 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:10:07.010 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128607 2024-09-12T08:10:07.010 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:10:07.010 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3800000418 ]] 2024-09-12T08:10:07.011 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3800000418 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\3\4\3\4\7\4\7 ]] 2024-09-12T08:10:07.011 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3800000418 2024-09-12T08:10:09.074 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3800000418 until 2024-09-12T09:10:08.320134+0000 (3600 sec) 2024-09-12T08:10:09.109 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:10:09.113 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3800000418 2024-09-12T08:10:09.890 INFO:tasks.workunit.client.0.smithi137.stderr:listed 11 entries 2024-09-12T08:10:09.938 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:10:09.939 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:10:19.939 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:10:19.939 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:10:19.949 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:10:19.953 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:10:20.899 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:10:20.904 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:10:20.909 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128620 2024-09-12T08:10:20.909 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:10:20.909 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3800000418 ]] 2024-09-12T08:10:20.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3800000418 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\0\0\0\0\0\4\1\8 ]] 2024-09-12T08:10:20.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:10:30.918 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:10:30.918 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:10:30.924 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:10:30.932 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:10:31.918 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:10:31.918 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:10:31.919 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128631 2024-09-12T08:10:31.919 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:10:31.919 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3800000418 ]] 2024-09-12T08:10:31.919 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3800000418 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\0\0\0\0\0\4\1\8 ]] 2024-09-12T08:10:31.919 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:10:41.930 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:10:41.930 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:10:41.930 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:10:41.940 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:10:42.941 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:10:42.942 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:10:42.943 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128642 2024-09-12T08:10:42.943 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:10:42.943 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3800000418 ]] 2024-09-12T08:10:42.943 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3800000418 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\0\0\0\0\0\4\1\8 ]] 2024-09-12T08:10:42.943 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:10:50.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:10:50.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:10:50.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:10:50.737 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:10:50.738 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:10:50.736+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:10:52.948 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:10:52.948 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:10:52.948 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:10:52.950 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:10:53.915 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:10:53.916 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:10:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128653 2024-09-12T08:10:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:10:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3800000418 ]] 2024-09-12T08:10:53.917 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3800000418 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\0\0\0\0\0\4\1\8 ]] 2024-09-12T08:10:53.918 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:11:03.919 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:11:03.932 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:11:03.934 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:11:03.936 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:11:04.909 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3800000418 2024-09-12T08:11:04.909 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:11:04.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128664 2024-09-12T08:11:04.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:11:04.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3800000418 ]] 2024-09-12T08:11:04.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3800000418 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\0\0\0\0\0\4\1\8 ]] 2024-09-12T08:11:04.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:11:14.912 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:11:14.914 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:11:14.921 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:11:14.927 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:11:15.840 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:11:15.841 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:11:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128675 2024-09-12T08:11:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:11:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2294234594 ]] 2024-09-12T08:11:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2294234594 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\0\0\0\0\0\4\1\8 ]] 2024-09-12T08:11:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2294234594 2024-09-12T08:11:18.108 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2294234594 until 2024-09-12T09:11:17.373756+0000 (3600 sec) 2024-09-12T08:11:18.135 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:11:18.135 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2294234594 2024-09-12T08:11:18.898 INFO:tasks.workunit.client.0.smithi137.stderr:listed 12 entries 2024-09-12T08:11:18.926 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:11:18.926 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:11:28.938 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:11:28.948 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:11:28.950 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:11:28.952 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:11:29.876 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:11:29.880 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:11:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128689 2024-09-12T08:11:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:11:29.881 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2294234594 ]] 2024-09-12T08:11:29.882 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2294234594 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\9\4\2\3\4\5\9\4 ]] 2024-09-12T08:11:29.882 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:11:30.436 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:11:30.435+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:11:30.436 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:11:30.435+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:11:39.884 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:11:39.884 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:11:39.884 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:11:39.884 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:11:40.873 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:11:40.873 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:11:40.874 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128700 2024-09-12T08:11:40.874 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:11:40.874 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2294234594 ]] 2024-09-12T08:11:40.874 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2294234594 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\9\4\2\3\4\5\9\4 ]] 2024-09-12T08:11:40.874 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:11:50.876 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:11:50.877 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:11:50.877 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:11:50.886 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:11:51.852 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:11:51.853 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:11:51.853 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128711 2024-09-12T08:11:51.853 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:11:51.853 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2294234594 ]] 2024-09-12T08:11:51.853 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2294234594 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\9\4\2\3\4\5\9\4 ]] 2024-09-12T08:11:51.853 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:12:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:11:59.998+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:11:59.998+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b91080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.002+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.002+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b90900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.006+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.006+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.018+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.018+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b90280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.022+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.022+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.026+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.026+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.030+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.030+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.034+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.034+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.038+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.038+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.042+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.042+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.046+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.046+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.050+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.050+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.054+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.054+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.054+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.054+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.066+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.066+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.074+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.074+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.078+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.078+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c65f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.094+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.094+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfa180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.101 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.102+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.101 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.102+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfa380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.106+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:00.108 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:12:00.106+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfa580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:12:01.862 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:12:01.862 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:12:01.862 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:12:01.865 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:12:02.706 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:12:02.706 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:12:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128722 2024-09-12T08:12:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:12:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2294234594 ]] 2024-09-12T08:12:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2294234594 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\9\4\2\3\4\5\9\4 ]] 2024-09-12T08:12:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:12:12.709 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:12:12.709 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:12:12.713 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:12:12.716 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:12:13.624 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2294234594 2024-09-12T08:12:13.624 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:12:13.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128733 2024-09-12T08:12:13.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:12:13.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2294234594 ]] 2024-09-12T08:12:13.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2294234594 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\9\4\2\3\4\5\9\4 ]] 2024-09-12T08:12:13.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:12:23.635 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:12:23.635 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:12:23.636 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:12:23.644 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:12:24.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:12:24.632 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:12:24.636 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128744 2024-09-12T08:12:24.636 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:12:24.636 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3572572957 ]] 2024-09-12T08:12:24.637 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3572572957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\9\4\2\3\4\5\9\4 ]] 2024-09-12T08:12:24.637 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3572572957 2024-09-12T08:12:27.141 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3572572957 until 2024-09-12T09:12:26.446997+0000 (3600 sec) 2024-09-12T08:12:27.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:12:27.183 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3572572957 2024-09-12T08:12:28.027 INFO:tasks.workunit.client.0.smithi137.stderr:listed 13 entries 2024-09-12T08:12:28.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:12:28.067 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:12:38.079 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:12:38.085 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:12:38.086 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:12:38.087 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:12:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:12:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:12:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128759 2024-09-12T08:12:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:12:39.116 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3572572957 ]] 2024-09-12T08:12:39.116 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3572572957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\7\2\5\7\2\9\5\7 ]] 2024-09-12T08:12:39.116 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:12:40.687 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:12:40.689+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:12:40.688 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:12:40.689+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:12:40.689 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:12:40.693+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:12:49.124 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:12:49.124 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:12:49.126 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:12:49.126 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:12:50.040 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:12:50.041 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:12:50.049 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128770 2024-09-12T08:12:50.049 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:12:50.049 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3572572957 ]] 2024-09-12T08:12:50.050 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3572572957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\7\2\5\7\2\9\5\7 ]] 2024-09-12T08:12:50.050 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.001+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.001+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e910d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.009+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.009+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e89c7500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.013+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.013+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e89c7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.017+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.017+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e89c7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.017+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.017+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.025+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.025+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98cfc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.029+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.029+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9943f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.033+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.033+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e965fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.037+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.037+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e98ced80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.045+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.045+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9943b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.045+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.045+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9943700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9943e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.057+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.057+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b90380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.061+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.061+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b91880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.060 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:13:00.060 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:13:00.060 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:13:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.065+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.065+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b90580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.069 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:13:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.077+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.077+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfa180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.085+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.085+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfa780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.093+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.093+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfa880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.097+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.097+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfb180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.105 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.109+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:00.105 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:00.109+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfb000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:13:01.000 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:13:01.000 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:13:01.001 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128781 2024-09-12T08:13:01.001 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:13:01.001 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3572572957 ]] 2024-09-12T08:13:01.001 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3572572957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\7\2\5\7\2\9\5\7 ]] 2024-09-12T08:13:01.001 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:11.004 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:13:11.006 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:13:11.008 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:13:11.021 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:13:12.002 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:13:12.002 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:13:12.002 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128792 2024-09-12T08:13:12.002 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:13:12.002 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3572572957 ]] 2024-09-12T08:13:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3572572957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\7\2\5\7\2\9\5\7 ]] 2024-09-12T08:13:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:22.017 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:13:22.019 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:13:22.023 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:13:22.033 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:13:22.968 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3572572957 2024-09-12T08:13:22.968 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:13:22.969 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128802 2024-09-12T08:13:22.969 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:13:22.969 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3572572957 ]] 2024-09-12T08:13:22.969 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3572572957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\7\2\5\7\2\9\5\7 ]] 2024-09-12T08:13:22.969 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:23.884 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:13:23.884+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:13:23.884 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:13:23.884+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:13:32.974 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:13:32.974 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:13:32.983 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:13:33.002 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:13:33.900 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:13:33.900 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:13:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128813 2024-09-12T08:13:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:13:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3644346538 ]] 2024-09-12T08:13:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3644346538 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\7\2\5\7\2\9\5\7 ]] 2024-09-12T08:13:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3644346538 2024-09-12T08:13:36.132 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3644346538 until 2024-09-12T09:13:35.378156+0000 (3600 sec) 2024-09-12T08:13:36.175 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:13:36.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3644346538 2024-09-12T08:13:36.957 INFO:tasks.workunit.client.0.smithi137.stderr:listed 14 entries 2024-09-12T08:13:36.983 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:13:36.983 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:40.093 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.088+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:13:40.097 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:40.096+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:13:44.362 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:13:44.364+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:13:46.985 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:13:46.987 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:13:46.992 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:13:46.995 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:13:48.053 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:13:48.053 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:13:48.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128828 2024-09-12T08:13:48.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:13:48.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3644346538 ]] 2024-09-12T08:13:48.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3644346538 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\6\4\4\3\4\6\5\3\8 ]] 2024-09-12T08:13:48.055 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:13:57.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:13:57.911 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:13:57.911 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:13:57.911 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:13:57.907+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:13:58.062 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:13:58.063 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:13:58.064 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:13:58.067 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:13:59.252 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:13:59.260 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:13:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128839 2024-09-12T08:13:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:13:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3644346538 ]] 2024-09-12T08:13:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3644346538 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\6\4\4\3\4\6\5\3\8 ]] 2024-09-12T08:13:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:13:59.999 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:59.999+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:13:59.999+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9972a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e910cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e90ef680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.019+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.019+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e90ee780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9c65600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.035+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.039+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfb180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.047+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9c65380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfbc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfbe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.091+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.091+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.091+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.092 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.091+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.095+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.095+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.095+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.095+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.099+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:00.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:14:00.099+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbf000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:14:08.529 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:08.523+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:14:08.530 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:08.527+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:14:09.329 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:14:09.329 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:14:09.329 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:14:09.334 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:14:10.354 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:14:10.355 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:14:10.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128850 2024-09-12T08:14:10.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:14:10.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3644346538 ]] 2024-09-12T08:14:10.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3644346538 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\6\4\4\3\4\6\5\3\8 ]] 2024-09-12T08:14:10.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:14:11.220 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.219+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:14:11.221 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.219+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:14:11.221 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.219+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:14:11.221 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:14:11.222 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:14:11.223+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:14:20.366 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:14:20.369 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:14:20.369 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:14:20.377 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:14:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:14:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:14:21.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128861 2024-09-12T08:14:21.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:14:21.584 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3644346538 ]] 2024-09-12T08:14:21.584 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3644346538 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\6\4\4\3\4\6\5\3\8 ]] 2024-09-12T08:14:21.584 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:14:31.587 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:14:31.588 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:14:31.594 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:14:31.601 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:14:32.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3644346538 2024-09-12T08:14:32.582 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:14:32.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128872 2024-09-12T08:14:32.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:14:32.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3644346538 ]] 2024-09-12T08:14:32.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3644346538 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\6\4\4\3\4\6\5\3\8 ]] 2024-09-12T08:14:32.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:14:42.584 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:14:42.585 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:14:42.585 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:14:42.586 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:14:43.866 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:14:43.869 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:14:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128883 2024-09-12T08:14:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:14:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:14:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\6\4\4\3\4\6\5\3\8 ]] 2024-09-12T08:14:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3549391526 2024-09-12T08:14:46.161 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3549391526 until 2024-09-12T09:14:45.435158+0000 (3600 sec) 2024-09-12T08:14:46.199 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3549391526 2024-09-12T08:14:46.202 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:14:46.997 INFO:tasks.workunit.client.0.smithi137.stderr:listed 15 entries 2024-09-12T08:14:47.034 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:14:47.034 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:14:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:14:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:14:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:14:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:14:58.053 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:14:58.054 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:14:58.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128898 2024-09-12T08:14:58.054 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:14:58.055 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:14:58.055 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:14:58.055 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:14:59.999 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.002+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.000 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.002+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e886d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.001 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.002+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e899b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.006+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95cb000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.010+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.010+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e899b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.014+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.014+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95cb300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.018+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.018+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9972400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.022+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.022+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b91f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.038+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.038+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9634e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.042+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.042+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9973b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.046+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.046+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e910cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.054+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.054+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9c64c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.058+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.058+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b90000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.062+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.062+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.070+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.070+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.074+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.074+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e89c6480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.078+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.078+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbf400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.090+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.090+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbf680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.094+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.094+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbf880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.094+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.098+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbfa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.098+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:15:00.098+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbfc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:15:08.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:15:08.010+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:15:08.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:15:08.010+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:15:08.063 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:15:08.063 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:15:08.064 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:15:08.064 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:15:09.160 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:15:09.160 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:15:09.161 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128909 2024-09-12T08:15:09.161 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:15:09.161 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:15:09.161 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:15:09.161 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:15:19.175 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:15:19.178 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:15:19.183 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:15:19.184 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:15:20.222 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:15:20.225 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:15:20.226 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128920 2024-09-12T08:15:20.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:15:20.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:15:20.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:15:20.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:15:24.357 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:15:24.357+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:15:30.231 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:15:30.231 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:15:30.234 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:15:30.244 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:15:31.177 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:15:31.177 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:15:31.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128931 2024-09-12T08:15:31.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:15:31.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:15:31.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:15:31.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:15:41.180 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:15:41.180 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:15:41.186 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:15:41.186 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:15:42.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:15:42.104 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:15:42.105 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128942 2024-09-12T08:15:42.105 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:15:42.105 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:15:42.105 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:15:42.105 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:15:52.106 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:15:52.107 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:15:52.107 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:15:52.113 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:15:53.136 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3549391526 2024-09-12T08:15:53.136 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:15:53.137 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128953 2024-09-12T08:15:53.137 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:15:53.137 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3549391526 ]] 2024-09-12T08:15:53.137 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3549391526 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:15:53.137 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:16:03.147 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:16:03.147 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:16:03.155 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:16:03.156 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:16:04.113 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:16:04.114 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:16:04.114 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128964 2024-09-12T08:16:04.114 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:16:04.115 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2595000226 ]] 2024-09-12T08:16:04.115 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2595000226 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\5\4\9\3\9\1\5\2\6 ]] 2024-09-12T08:16:04.115 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2595000226 2024-09-12T08:16:05.207 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:05.209+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9fd3200 image watch failed: 94042233420544, (107) Transport endpoint is not connected 2024-09-12T08:16:05.207 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:05.209+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9fd3200 handle_error: handle=94042233420544: (107) Transport endpoint is not connected 2024-09-12T08:16:05.208 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:05.209+0000 7f9d2531b640 -1 librbd::watcher::RewatchRequest: 0x5587e93defa0 handle_unwatch client blocklisted 2024-09-12T08:16:05.208 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:05.209+0000 7f9d2531b640 -1 librbd::Watcher: 0x5587e9fd3200 handle_rewatch: client blocklisted 2024-09-12T08:16:06.208 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2595000226 until 2024-09-12T09:16:05.433594+0000 (3600 sec) 2024-09-12T08:16:06.220 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.220 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904ed00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.220 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9b32f00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.221 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.221 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e94e2280 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.221 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587e9d751c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.221 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:16:06.221+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e97df200 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:16:06.255 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:16:06.255 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2595000226 2024-09-12T08:16:07.013 INFO:tasks.workunit.client.0.smithi137.stderr:listed 16 entries 2024-09-12T08:16:07.044 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:16:07.044 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:16:17.049 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:16:17.050 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:16:17.054 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:16:17.062 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:16:18.145 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:16:18.145 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:16:18.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128978 2024-09-12T08:16:18.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:16:18.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2595000226 ]] 2024-09-12T08:16:18.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2595000226 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\9\5\0\0\0\2\2\6 ]] 2024-09-12T08:16:18.146 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:16:28.149 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:16:28.149 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:16:28.150 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:16:28.168 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:16:29.093 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:16:29.100 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:16:29.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726128989 2024-09-12T08:16:29.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:16:29.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2595000226 ]] 2024-09-12T08:16:29.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2595000226 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\9\5\0\0\0\2\2\6 ]] 2024-09-12T08:16:29.105 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:16:31.198 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.192+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:16:31.198 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.192+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:16:31.204 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:16:31.204 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:16:31.204 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:16:31.204 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:16:31.205 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:16:31.207 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:16:31.207 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:16:31.207 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:16:31.207 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:16:31.207 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:16:31.207 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:31.204+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:16:39.106 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:16:39.106 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:16:39.122 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:16:39.131 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:16:40.290 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:16:40.290 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:16:40.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129000 2024-09-12T08:16:40.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:16:40.292 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2595000226 ]] 2024-09-12T08:16:40.292 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2595000226 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\9\5\0\0\0\2\2\6 ]] 2024-09-12T08:16:40.292 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:16:50.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:16:50.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:16:50.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:16:50.295 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:16:51.220 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:16:51.229 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:16:51.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129011 2024-09-12T08:16:51.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:16:51.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2595000226 ]] 2024-09-12T08:16:51.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2595000226 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\9\5\0\0\0\2\2\6 ]] 2024-09-12T08:16:51.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:16:58.844 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:16:58.845 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:16:58.844+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:17:01.232 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:17:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:17:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:17:01.252 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:17:02.240 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2595000226 2024-09-12T08:17:02.240 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:17:02.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129022 2024-09-12T08:17:02.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:17:02.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2595000226 ]] 2024-09-12T08:17:02.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2595000226 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\9\5\0\0\0\2\2\6 ]] 2024-09-12T08:17:02.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:17:12.245 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:17:12.258 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:17:12.267 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:17:12.275 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:17:13.281 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:17:13.286 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:17:13.287 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129033 2024-09-12T08:17:13.287 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:17:13.287 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1486950863 ]] 2024-09-12T08:17:13.287 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1486950863 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\9\5\0\0\0\2\2\6 ]] 2024-09-12T08:17:13.288 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1486950863 2024-09-12T08:17:15.157 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1486950863 until 2024-09-12T09:17:14.364447+0000 (3600 sec) 2024-09-12T08:17:15.187 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:17:15.187 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1486950863 2024-09-12T08:17:15.318 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:17:15.319+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:17:15.318 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:17:15.319+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:17:15.899 INFO:tasks.workunit.client.0.smithi137.stderr:listed 17 entries 2024-09-12T08:17:15.925 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:17:15.925 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:17:16.533 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:17:16.523+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:17:16.533 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:17:16.523+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:17:16.533 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:17:16.523+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:17:16.533 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:17:16.523+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:17:25.927 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:17:25.936 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:17:25.938 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:17:25.946 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:17:26.844 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:17:26.844 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:17:26.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129046 2024-09-12T08:17:26.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:17:26.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1486950863 ]] 2024-09-12T08:17:26.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1486950863 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\8\6\9\5\0\8\6\3 ]] 2024-09-12T08:17:26.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:17:36.851 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:17:36.852 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:17:36.853 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:17:36.857 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:17:37.858 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:17:37.871 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:17:37.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129057 2024-09-12T08:17:37.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:17:37.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1486950863 ]] 2024-09-12T08:17:37.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1486950863 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\8\6\9\5\0\8\6\3 ]] 2024-09-12T08:17:37.872 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:17:47.874 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:17:47.877 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:17:47.887 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:17:47.899 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:17:48.910 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:17:48.910 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:17:48.911 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129068 2024-09-12T08:17:48.911 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:17:48.912 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1486950863 ]] 2024-09-12T08:17:48.912 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1486950863 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\8\6\9\5\0\8\6\3 ]] 2024-09-12T08:17:48.912 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:17:52.009 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:17:52.002+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:17:52.009 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:17:52.002+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:17:58.912 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:17:58.918 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:17:58.923 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:17:58.931 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:18:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.006+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b91900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.006+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.006+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9c64900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.010+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.010+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e90eea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.010+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.010+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9c64c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.014+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.014+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9721e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.018+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.018+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f52d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.026+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.026+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f52900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.030+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.030+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f52700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.030+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.030+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.034+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.034+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.038+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.038+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f52d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.054+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.054+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f52f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.066+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.066+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.070+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.070+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.070+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.070+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.074+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.074+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.078+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.078+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.086+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.086+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f53e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.086+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.086+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d58080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.090+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:18:00.090+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d58280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:18:00.093 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:18:00.094 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:18:00.095 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129080 2024-09-12T08:18:00.095 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:18:00.095 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1486950863 ]] 2024-09-12T08:18:00.095 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1486950863 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\8\6\9\5\0\8\6\3 ]] 2024-09-12T08:18:00.095 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:18:00.466 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:18:00.466+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:18:00.466 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:18:00.466+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:18:04.313 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:04.314+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:18:04.314 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:04.314+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:18:04.314 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:04.314+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:18:04.314 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:04.314+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:18:04.315 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:04.318+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:18:10.104 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:18:10.104 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:18:10.109 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:18:10.110 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:18:11.082 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1486950863 2024-09-12T08:18:11.082 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:18:11.083 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129091 2024-09-12T08:18:11.084 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:18:11.084 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1486950863 ]] 2024-09-12T08:18:11.084 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1486950863 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\8\6\9\5\0\8\6\3 ]] 2024-09-12T08:18:11.084 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:18:21.087 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:18:21.093 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:18:21.099 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:18:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:18:22.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:18:22.118 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:18:22.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129102 2024-09-12T08:18:22.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:18:22.118 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3839580397 ]] 2024-09-12T08:18:22.119 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3839580397 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\8\6\9\5\0\8\6\3 ]] 2024-09-12T08:18:22.119 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3839580397 2024-09-12T08:18:23.837 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3839580397 until 2024-09-12T09:18:23.059940+0000 (3600 sec) 2024-09-12T08:18:23.890 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3839580397 2024-09-12T08:18:23.899 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:18:24.636 INFO:tasks.workunit.client.0.smithi137.stderr:listed 18 entries 2024-09-12T08:18:24.662 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:18:24.663 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:18:34.664 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:18:34.665 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:18:34.670 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:18:34.678 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:18:35.572 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:18:35.572 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:18:35.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129115 2024-09-12T08:18:35.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:18:35.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3839580397 ]] 2024-09-12T08:18:35.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3839580397 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\3\9\5\8\0\3\9\7 ]] 2024-09-12T08:18:35.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:18:37.288 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:37.289+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:18:45.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:18:45.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:18:45.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:18:45.582 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:18:46.552 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:18:46.552 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:18:46.552 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129126 2024-09-12T08:18:46.552 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:18:46.552 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3839580397 ]] 2024-09-12T08:18:46.553 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3839580397 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\3\9\5\8\0\3\9\7 ]] 2024-09-12T08:18:46.553 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:18:55.032 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.032+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:18:55.037 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:18:55.037 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:18:55.038 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.036+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:18:55.128 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.124+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:18:55.139 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:18:55.140 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:18:55.140 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:18:55.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:18:56.556 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:18:56.557 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:18:56.571 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:18:56.580 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:18:57.523 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:18:57.523 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:18:57.523 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129137 2024-09-12T08:18:57.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:18:57.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3839580397 ]] 2024-09-12T08:18:57.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3839580397 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\3\9\5\8\0\3\9\7 ]] 2024-09-12T08:18:57.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:19:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9943780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e992c000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.004+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.004+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e95a8d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e886c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e89c6b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.016+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f53a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.020+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.020+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9c64a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.024+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.024+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.028+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.028+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfbb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.032+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfb700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.036+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.036+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e944f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.036+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.036+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9c64f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.040+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.040+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9973680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.044+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.044+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfbf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.048+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.048+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfb100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.048+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.048+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d58300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.052+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.052+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cbe000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.056+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.056+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cbf180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.056+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.056+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d58c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.060+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:19:00.060+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d58a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:19:07.529 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:19:07.529 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:19:07.533 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:19:07.542 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:19:08.514 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:19:08.514 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:19:08.514 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129148 2024-09-12T08:19:08.514 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:19:08.514 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3839580397 ]] 2024-09-12T08:19:08.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3839580397 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\3\9\5\8\0\3\9\7 ]] 2024-09-12T08:19:08.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:19:18.524 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:19:18.524 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:19:18.524 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:19:18.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:19:19.444 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3839580397 2024-09-12T08:19:19.444 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:19:19.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129159 2024-09-12T08:19:19.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:19:19.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3839580397 ]] 2024-09-12T08:19:19.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3839580397 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\3\9\5\8\0\3\9\7 ]] 2024-09-12T08:19:19.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:19:29.448 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:19:29.448 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:19:29.451 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:19:29.453 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:19:30.667 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:19:30.668 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:19:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129170 2024-09-12T08:19:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:19:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1429115853 ]] 2024-09-12T08:19:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1429115853 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\8\3\9\5\8\0\3\9\7 ]] 2024-09-12T08:19:30.669 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1429115853 2024-09-12T08:19:33.184 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1429115853 until 2024-09-12T09:19:32.429290+0000 (3600 sec) 2024-09-12T08:19:33.233 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:19:33.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1429115853 2024-09-12T08:19:34.035 INFO:tasks.workunit.client.0.smithi137.stderr:listed 19 entries 2024-09-12T08:19:34.073 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:19:34.073 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:19:44.075 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:19:44.075 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:19:44.083 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:19:44.093 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:19:45.074 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:19:45.074 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:19:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129185 2024-09-12T08:19:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:19:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1429115853 ]] 2024-09-12T08:19:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1429115853 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\2\9\1\1\5\8\5\3 ]] 2024-09-12T08:19:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:19:51.964 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:19:51.963+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:19:51.968 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:19:51.963+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:19:51.969 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:19:51.963+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:19:51.969 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:19:51.963+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:19:55.077 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:19:55.077 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:19:55.079 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:19:55.092 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:19:56.072 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:19:56.073 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:19:56.073 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129196 2024-09-12T08:19:56.073 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:19:56.074 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1429115853 ]] 2024-09-12T08:19:56.074 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1429115853 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\2\9\1\1\5\8\5\3 ]] 2024-09-12T08:19:56.074 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:20:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.003+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9c64600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.007+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.007+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e89c6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.011+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.011+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9973b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.019+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.019+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e89c6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.023+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.023+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbe300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.031+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.031+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b91700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.035+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.035+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d58900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.039+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.043+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d58380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.047+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.047+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.051+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.051+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.059+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.059+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d58e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.063+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.063+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbf100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.067+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.067+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.075+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.075+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.079+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.079+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.083+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.083+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.087+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.087+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.092 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.091+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.092 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.091+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.095+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.095+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.099+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:00.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:20:00.099+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9616200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:20:06.080 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:20:06.080 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:20:06.080 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:20:06.096 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:20:07.101 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:20:07.101 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:20:07.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129207 2024-09-12T08:20:07.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:20:07.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1429115853 ]] 2024-09-12T08:20:07.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1429115853 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\2\9\1\1\5\8\5\3 ]] 2024-09-12T08:20:07.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:20:17.104 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:20:17.107 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:20:17.127 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:20:17.132 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:20:18.020 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:20:18.020 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:20:18.021 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129218 2024-09-12T08:20:18.021 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:20:18.021 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1429115853 ]] 2024-09-12T08:20:18.021 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1429115853 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\2\9\1\1\5\8\5\3 ]] 2024-09-12T08:20:18.021 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:20:18.382 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:18.378+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:20:18.383 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:18.378+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:20:18.393 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:18.386+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:20:18.394 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:18.386+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:20:18.394 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:18.386+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:20:28.023 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:20:28.024 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:20:28.025 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:20:28.029 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:20:28.921 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1429115853 2024-09-12T08:20:28.921 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:20:28.922 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129228 2024-09-12T08:20:28.922 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:20:28.922 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1429115853 ]] 2024-09-12T08:20:28.922 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1429115853 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\2\9\1\1\5\8\5\3 ]] 2024-09-12T08:20:28.922 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:20:38.924 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:20:38.928 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:20:38.929 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:20:38.934 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:20:39.882 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:20:39.883 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:20:39.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129239 2024-09-12T08:20:39.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:20:39.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2206968216 ]] 2024-09-12T08:20:39.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2206968216 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\2\9\1\1\5\8\5\3 ]] 2024-09-12T08:20:39.883 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2206968216 2024-09-12T08:20:41.611 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2206968216 until 2024-09-12T09:20:40.839075+0000 (3600 sec) 2024-09-12T08:20:41.666 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:20:41.666 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2206968216 2024-09-12T08:20:41.781 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:41.774+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:20:41.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:41.778+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:20:41.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:41.778+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:20:41.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:20:41.778+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:20:42.499 INFO:tasks.workunit.client.0.smithi137.stderr:listed 20 entries 2024-09-12T08:20:42.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:20:42.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:20:52.529 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:20:52.529 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:20:52.535 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:20:52.544 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:20:53.475 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:20:53.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:20:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129253 2024-09-12T08:20:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:20:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2206968216 ]] 2024-09-12T08:20:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2206968216 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\0\6\9\6\8\2\1\6 ]] 2024-09-12T08:20:53.485 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:21:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.002+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.002+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.006+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.010+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.010+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.014+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.014+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.018+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.018+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.022+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.022+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.030+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.030+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.030+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.030+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.038+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.038+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.042+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.042+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.042+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.046+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.050+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.050+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.050+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.050+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.054+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.054+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.058+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.058+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.062+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.062+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.066+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.066+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.070+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.070+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.074+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.074+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.078+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:21:00.078+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:21:03.498 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:21:03.499 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:21:03.499 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:21:03.507 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:21:04.574 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:21:04.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:21:04.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129264 2024-09-12T08:21:04.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:21:04.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2206968216 ]] 2024-09-12T08:21:04.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2206968216 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\0\6\9\6\8\2\1\6 ]] 2024-09-12T08:21:04.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:21:14.585 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:21:14.585 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:21:14.585 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:21:14.590 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:21:15.534 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:21:15.534 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:21:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129275 2024-09-12T08:21:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:21:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2206968216 ]] 2024-09-12T08:21:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2206968216 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\0\6\9\6\8\2\1\6 ]] 2024-09-12T08:21:15.535 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:21:25.537 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:21:25.549 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:21:25.552 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:21:25.561 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:21:26.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:21:26.542 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:21:26.543 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129286 2024-09-12T08:21:26.543 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:21:26.543 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2206968216 ]] 2024-09-12T08:21:26.543 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2206968216 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\0\6\9\6\8\2\1\6 ]] 2024-09-12T08:21:26.544 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:21:36.484 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:21:36.481+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:21:36.484 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:21:36.481+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:21:36.546 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:21:36.546 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:21:36.546 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:21:36.549 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:21:37.530 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2206968216 2024-09-12T08:21:37.531 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:21:37.531 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129297 2024-09-12T08:21:37.531 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:21:37.532 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2206968216 ]] 2024-09-12T08:21:37.532 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2206968216 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\0\6\9\6\8\2\1\6 ]] 2024-09-12T08:21:37.532 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:21:47.533 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:21:47.533 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:21:47.537 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:21:47.538 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:21:47.656 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:21:47.653+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:21:47.660 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:21:47.657+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:21:47.671 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:21:47.661+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:21:47.671 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:21:47.661+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:21:48.668 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:21:48.677 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:21:48.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129308 2024-09-12T08:21:48.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:21:48.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2188351739 ]] 2024-09-12T08:21:48.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2188351739 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\0\6\9\6\8\2\1\6 ]] 2024-09-12T08:21:48.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2188351739 2024-09-12T08:21:51.250 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2188351739 until 2024-09-12T09:21:50.479217+0000 (3600 sec) 2024-09-12T08:21:51.301 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:21:51.314 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2188351739 2024-09-12T08:21:52.244 INFO:tasks.workunit.client.0.smithi137.stderr:listed 21 entries 2024-09-12T08:21:52.298 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:21:52.299 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:22:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95ca500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d58200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d59580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d59880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.016+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.024+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.024+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.024+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.024+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f53700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.032+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.040+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.040+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.048+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.048+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.052+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.052+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfa800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.060+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.060+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.068+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.068+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.076+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.076+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.084+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.084+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.084+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9ef3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.084+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.088+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.088+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.092+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.092+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9ef3c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.096+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:00.096+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:22:02.305 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:22:02.341 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:22:02.341 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:22:02.341 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:22:03.258 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:22:03.258 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:22:03.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129323 2024-09-12T08:22:03.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:22:03.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2188351739 ]] 2024-09-12T08:22:03.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2188351739 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\8\8\3\5\1\7\3\9 ]] 2024-09-12T08:22:03.259 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:22:05.049 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:05.040+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:22:05.050 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:05.044+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:22:05.050 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:05.044+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:22:05.053 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:05.052+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:22:13.263 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:22:13.263 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:22:13.267 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:22:13.269 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:22:14.338 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:22:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:22:14.340 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129334 2024-09-12T08:22:14.341 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:22:14.341 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2188351739 ]] 2024-09-12T08:22:14.341 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2188351739 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\8\8\3\5\1\7\3\9 ]] 2024-09-12T08:22:14.341 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:22:20.066 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:20.060+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:22:20.073 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:20.072+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:22:20.073 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:20.072+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:22:20.073 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:22:20.072+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:22:24.348 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:22:24.348 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:22:24.348 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:22:24.352 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:22:25.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:22:25.420 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:22:25.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129345 2024-09-12T08:22:25.421 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:22:25.421 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2188351739 ]] 2024-09-12T08:22:25.421 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2188351739 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\8\8\3\5\1\7\3\9 ]] 2024-09-12T08:22:25.421 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:22:35.423 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:22:35.424 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:22:35.426 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:22:35.426 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:22:36.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:22:36.448 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:22:36.449 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129356 2024-09-12T08:22:36.449 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:22:36.449 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2188351739 ]] 2024-09-12T08:22:36.449 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2188351739 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\8\8\3\5\1\7\3\9 ]] 2024-09-12T08:22:36.450 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:22:38.635 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:22:38.632+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:22:46.451 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:22:46.461 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:22:46.465 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:22:46.469 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:22:47.490 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2188351739 2024-09-12T08:22:47.490 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:22:47.490 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129367 2024-09-12T08:22:47.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:22:47.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2188351739 ]] 2024-09-12T08:22:47.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2188351739 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\8\8\3\5\1\7\3\9 ]] 2024-09-12T08:22:47.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:22:57.493 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:22:57.495 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:22:57.497 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:22:57.510 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:22:58.406 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:22:58.406 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:22:58.407 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129378 2024-09-12T08:22:58.407 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:22:58.407 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:22:58.407 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\8\8\3\5\1\7\3\9 ]] 2024-09-12T08:22:58.407 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/4051686922 2024-09-12T08:23:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:59.999+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:22:59.999+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e899bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.003+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.003+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e95ca500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.003+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.003+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.007+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.007+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.011+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.011+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.015+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.015+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.019+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.019+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.027+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.027+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.043+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.043+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.043+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.043+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.047+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.047+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.051+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.051+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.051+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.051+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.055+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.055+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.059+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.059+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.067+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.067+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.067+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.067+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.067+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.067+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f21f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.071+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f26180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:23:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f26380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:23:00.252 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/4051686922 until 2024-09-12T09:22:59.445922+0000 (3600 sec) 2024-09-12T08:23:00.312 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:23:00.315 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/4051686922 2024-09-12T08:23:01.048 INFO:tasks.workunit.client.0.smithi137.stderr:listed 22 entries 2024-09-12T08:23:01.084 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:23:01.084 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:23:08.834 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:08.831+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:23:08.844 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:08.839+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:23:11.090 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:23:11.090 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:23:11.100 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:23:11.108 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:23:12.080 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:23:12.080 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:23:12.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129392 2024-09-12T08:23:12.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:23:12.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:23:12.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:23:12.081 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:23:17.715 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:17.711+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:23:17.715 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:17.711+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:23:17.715 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:17.711+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:23:22.083 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:23:22.083 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:23:22.085 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:23:22.090 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:23:23.098 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:23:23.099 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:23:23.100 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129403 2024-09-12T08:23:23.100 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:23:23.100 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:23:23.101 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:23:23.101 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:23:33.106 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:23:33.110 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:23:33.110 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:23:33.118 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:23:34.109 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:23:34.109 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:23:34.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129414 2024-09-12T08:23:34.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:23:34.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:23:34.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:23:34.110 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:23:35.564 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:35.559+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:23:35.564 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:35.559+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:23:35.568 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:23:35.563+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:23:44.112 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:23:44.118 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:23:44.126 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:23:44.127 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:23:45.142 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:23:45.142 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:23:45.143 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129425 2024-09-12T08:23:45.143 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:23:45.143 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:23:45.143 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:23:45.144 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:23:55.146 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:23:55.146 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:23:55.146 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:23:55.148 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:23:56.209 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:23:56.210 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:23:56.210 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129436 2024-09-12T08:23:56.210 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:23:56.210 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:23:56.211 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:23:56.211 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:24:06.215 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:24:06.215 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:24:06.216 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:24:06.224 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:24:07.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4051686922 2024-09-12T08:24:07.170 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:24:07.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129447 2024-09-12T08:24:07.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:24:07.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4051686922 ]] 2024-09-12T08:24:07.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4051686922 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:24:07.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:24:17.178 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:24:17.179 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:24:17.179 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:24:17.180 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:24:18.155 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:24:18.156 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:24:18.157 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129458 2024-09-12T08:24:18.157 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:24:18.157 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1449742175 ]] 2024-09-12T08:24:18.157 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1449742175 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\1\6\8\6\9\2\2 ]] 2024-09-12T08:24:18.157 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1449742175 2024-09-12T08:24:20.309 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1449742175 until 2024-09-12T09:24:19.491092+0000 (3600 sec) 2024-09-12T08:24:20.324 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:24:20.322+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:24:20.324 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:24:20.322+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:24:20.380 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:24:20.380 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1449742175 2024-09-12T08:24:21.172 INFO:tasks.workunit.client.0.smithi137.stderr:listed 23 entries 2024-09-12T08:24:21.198 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:24:21.198 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:24:31.200 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:24:31.200 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:24:31.202 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:24:31.209 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:24:32.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:24:32.241 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:24:32.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129472 2024-09-12T08:24:32.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:24:32.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1449742175 ]] 2024-09-12T08:24:32.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1449742175 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\4\9\7\4\2\1\7\5 ]] 2024-09-12T08:24:32.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:24:42.244 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:24:42.244 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:24:42.249 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:24:42.249 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:24:43.270 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:24:43.270 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:24:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129483 2024-09-12T08:24:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:24:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1449742175 ]] 2024-09-12T08:24:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1449742175 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\4\9\7\4\2\1\7\5 ]] 2024-09-12T08:24:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:24:44.348 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:24:44.350+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:24:44.348 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:24:44.350+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:24:47.712 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:24:47.710+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:24:53.273 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:24:53.276 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:24:53.280 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:24:53.301 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:24:54.375 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:24:54.375 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:24:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129494 2024-09-12T08:24:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:24:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1449742175 ]] 2024-09-12T08:24:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1449742175 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\4\9\7\4\2\1\7\5 ]] 2024-09-12T08:24:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:25:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.001+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.001+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f26e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.005+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.009+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.009+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.013+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.013+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.013+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.013+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.017+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.017+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.021+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.021+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.025+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.025+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.029+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.029+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.033+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.033+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.041+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.041+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.045+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.045+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.049+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.049+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.057+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.057+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.077+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.077+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.081+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.081+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.085+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.085+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.093+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:25:00.093+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:25:04.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:25:04.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:25:04.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:25:04.396 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:25:05.446 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:25:05.446 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:25:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129505 2024-09-12T08:25:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:25:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1449742175 ]] 2024-09-12T08:25:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1449742175 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\4\9\7\4\2\1\7\5 ]] 2024-09-12T08:25:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:25:15.449 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:25:15.453 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:25:15.454 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:25:15.464 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:25:16.446 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1449742175 2024-09-12T08:25:16.446 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:25:16.447 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129516 2024-09-12T08:25:16.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:25:16.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1449742175 ]] 2024-09-12T08:25:16.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1449742175 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\4\9\7\4\2\1\7\5 ]] 2024-09-12T08:25:16.448 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:25:26.454 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:25:26.466 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:25:26.472 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:25:26.479 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:25:27.415 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:25:27.416 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:25:27.416 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129527 2024-09-12T08:25:27.416 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:25:27.416 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/666035049 ]] 2024-09-12T08:25:27.417 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/666035049 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\4\4\9\7\4\2\1\7\5 ]] 2024-09-12T08:25:27.417 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/666035049 2024-09-12T08:25:29.270 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/666035049 until 2024-09-12T09:25:28.510321+0000 (3600 sec) 2024-09-12T08:25:29.293 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/666035049 2024-09-12T08:25:29.294 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:25:30.211 INFO:tasks.workunit.client.0.smithi137.stderr:listed 24 entries 2024-09-12T08:25:30.253 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:25:30.253 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:25:40.259 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:25:40.261 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:25:40.264 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:25:40.264 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:25:41.181 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:25:41.182 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:25:41.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129541 2024-09-12T08:25:41.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:25:41.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/666035049 ]] 2024-09-12T08:25:41.183 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/666035049 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\6\6\0\3\5\0\4\9 ]] 2024-09-12T08:25:41.183 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:25:51.196 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:25:51.196 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:25:51.196 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:25:51.196 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:25:52.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:25:52.382 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:25:52.382 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129552 2024-09-12T08:25:52.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:25:52.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/666035049 ]] 2024-09-12T08:25:52.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/666035049 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\6\6\0\3\5\0\4\9 ]] 2024-09-12T08:25:52.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:26:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.000+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.000+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f26080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.016+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.020+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.020+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.032+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.040+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.040+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.048+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.048+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.064+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.064+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.068+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.068+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.072+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.072+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea093d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.084+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.084+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea093f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.088+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.088+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a4100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.088+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.088+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a4300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.092+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.092+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a4500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.096+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.096+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a4700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.096+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:26:00.096+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:26:02.384 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:26:02.385 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:26:02.385 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:26:02.386 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:26:03.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:26:03.498 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:26:03.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129563 2024-09-12T08:26:03.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:26:03.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/666035049 ]] 2024-09-12T08:26:03.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/666035049 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\6\6\0\3\5\0\4\9 ]] 2024-09-12T08:26:03.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:26:13.519 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:26:13.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:26:13.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:26:13.546 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:26:14.398 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:26:14.399 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:26:14.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129574 2024-09-12T08:26:14.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:26:14.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/666035049 ]] 2024-09-12T08:26:14.401 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/666035049 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\6\6\0\3\5\0\4\9 ]] 2024-09-12T08:26:14.401 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:26:24.404 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:26:24.404 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:26:24.404 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:26:24.420 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:26:25.456 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/666035049 2024-09-12T08:26:25.457 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:26:25.457 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129585 2024-09-12T08:26:25.457 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:26:25.457 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/666035049 ]] 2024-09-12T08:26:25.458 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/666035049 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\6\6\0\3\5\0\4\9 ]] 2024-09-12T08:26:25.458 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:26:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:26:35.461 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:26:35.472 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:26:35.477 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:26:36.432 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:26:36.432 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:26:36.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129596 2024-09-12T08:26:36.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:26:36.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2523539757 ]] 2024-09-12T08:26:36.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2523539757 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\6\6\0\3\5\0\4\9 ]] 2024-09-12T08:26:36.433 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2523539757 2024-09-12T08:26:38.295 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2523539757 until 2024-09-12T09:26:37.491989+0000 (3600 sec) 2024-09-12T08:26:38.326 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:26:38.327 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2523539757 2024-09-12T08:26:39.062 INFO:tasks.workunit.client.0.smithi137.stderr:listed 25 entries 2024-09-12T08:26:39.091 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:26:39.091 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:26:49.097 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:26:49.099 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:26:49.099 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:26:49.109 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:26:50.181 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:26:50.181 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:26:50.181 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129610 2024-09-12T08:26:50.181 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:26:50.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2523539757 ]] 2024-09-12T08:26:50.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2523539757 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\2\3\5\3\9\7\5\7 ]] 2024-09-12T08:26:50.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:26:51.377 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:51.368+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:26:51.385 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:51.380+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:26:51.385 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:51.380+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:26:51.385 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:51.380+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:26:52.013 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:26:52.014 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:26:52.014 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:26:52.014 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:26:52.014 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:26:52.014 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:52.008+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:26:56.355 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.352+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:26:56.359 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.356+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:26:56.360 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.360+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:26:56.366 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.364+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:26:56.370 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.368+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:26:56.370 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.368+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:26:56.370 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.368+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:26:56.370 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:26:56.368+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:27:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.000+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.000+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e90eec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.008+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.008+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbf480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.008+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbf900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.016+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbf880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.028+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.028+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbf600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.032+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.032+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b91500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.036+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.036+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d59780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.040+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.040+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d58100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.048+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.048+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.052+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.052+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.056+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.056+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d59280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.064+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.064+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d59b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.068+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.068+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.072+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.072+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.076+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.076+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.080+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.080+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.084+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.084+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95ca500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.092+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.092+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfaf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.096+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.096+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a4e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.108 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.108+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.108 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:00.108+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a4f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:27:00.183 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:27:00.183 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:27:00.184 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:27:00.189 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:27:01.248 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:27:01.248 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:27:01.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129621 2024-09-12T08:27:01.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:27:01.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2523539757 ]] 2024-09-12T08:27:01.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2523539757 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\2\3\5\3\9\7\5\7 ]] 2024-09-12T08:27:01.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:27:11.265 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:27:11.267 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:27:11.269 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:27:11.269 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:27:12.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:27:12.306 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:27:12.307 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129632 2024-09-12T08:27:12.307 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:27:12.307 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2523539757 ]] 2024-09-12T08:27:12.307 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2523539757 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\2\3\5\3\9\7\5\7 ]] 2024-09-12T08:27:12.307 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:27:22.309 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:27:22.309 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:27:22.314 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:27:22.315 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:27:23.356 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:27:23.357 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:27:23.358 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129643 2024-09-12T08:27:23.358 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:27:23.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2523539757 ]] 2024-09-12T08:27:23.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2523539757 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\2\3\5\3\9\7\5\7 ]] 2024-09-12T08:27:23.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:27:33.361 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:27:33.361 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:27:33.364 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:27:33.367 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:27:34.328 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2523539757 2024-09-12T08:27:34.328 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:27:34.329 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129654 2024-09-12T08:27:34.329 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:27:34.329 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2523539757 ]] 2024-09-12T08:27:34.329 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2523539757 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\2\3\5\3\9\7\5\7 ]] 2024-09-12T08:27:34.329 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:27:44.331 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:27:44.332 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:27:44.338 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:27:44.338 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:27:45.315 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:27:45.315 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:27:45.316 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129665 2024-09-12T08:27:45.316 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:27:45.316 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2111985957 ]] 2024-09-12T08:27:45.316 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2111985957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\2\3\5\3\9\7\5\7 ]] 2024-09-12T08:27:45.316 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2111985957 2024-09-12T08:27:47.126 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:47.123+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:27:47.126 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:47.123+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:27:47.126 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:47.123+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:27:47.335 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2111985957 until 2024-09-12T09:27:46.639408+0000 (3600 sec) 2024-09-12T08:27:47.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:27:47.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2111985957 2024-09-12T08:27:48.229 INFO:tasks.workunit.client.0.smithi137.stderr:listed 26 entries 2024-09-12T08:27:48.280 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:27:48.281 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:27:48.862 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:27:48.862 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:27:48.862 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:27:48.862 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:27:48.862 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.855+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.859+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.859+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.859+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.859+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:27:48.863 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:48.859+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:27:55.697 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:55.695+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:27:55.698 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:55.695+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:27:55.698 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:55.695+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:27:58.289 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:27:58.289 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:27:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:27:58.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:27:58.531 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:27:58.527+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:27:59.308 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:27:59.308 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:27:59.309 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129679 2024-09-12T08:27:59.309 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:27:59.309 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2111985957 ]] 2024-09-12T08:27:59.309 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2111985957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\1\1\9\8\5\9\5\7 ]] 2024-09-12T08:27:59.309 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:28:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:59.999+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:27:59.999+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f21500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea092780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.019+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.019+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.023+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.023+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea092f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.027+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.027+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f26800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.031+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.031+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f26d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.036 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.043+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.043+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.043+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.043+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.051+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.051+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a4a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a4c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:28:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea160180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:28:09.310 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:28:09.314 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:28:09.316 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:28:09.325 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:28:10.234 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:28:10.238 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:28:10.245 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129690 2024-09-12T08:28:10.245 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:28:10.245 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2111985957 ]] 2024-09-12T08:28:10.245 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2111985957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\1\1\9\8\5\9\5\7 ]] 2024-09-12T08:28:10.245 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:28:20.247 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:28:20.248 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:28:20.248 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:28:20.248 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:28:21.154 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:28:21.155 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:28:21.156 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129701 2024-09-12T08:28:21.156 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:28:21.156 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2111985957 ]] 2024-09-12T08:28:21.156 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2111985957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\1\1\9\8\5\9\5\7 ]] 2024-09-12T08:28:21.156 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:28:27.678 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:28:27.674+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:28:31.158 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:28:31.158 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:28:31.158 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:28:31.160 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:28:32.050 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:28:32.050 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:28:32.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129712 2024-09-12T08:28:32.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:28:32.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2111985957 ]] 2024-09-12T08:28:32.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2111985957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\1\1\9\8\5\9\5\7 ]] 2024-09-12T08:28:32.051 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:28:42.052 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:28:42.053 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:28:42.053 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:28:42.053 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:28:43.041 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2111985957 2024-09-12T08:28:43.041 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:28:43.042 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129723 2024-09-12T08:28:43.042 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:28:43.042 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2111985957 ]] 2024-09-12T08:28:43.042 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2111985957 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\1\1\9\8\5\9\5\7 ]] 2024-09-12T08:28:43.042 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:28:53.044 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:28:53.045 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:28:53.045 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:28:53.052 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:28:54.070 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:28:54.070 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:28:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129734 2024-09-12T08:28:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:28:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/113963503 ]] 2024-09-12T08:28:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/113963503 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\1\1\1\9\8\5\9\5\7 ]] 2024-09-12T08:28:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/113963503 2024-09-12T08:28:56.347 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/113963503 until 2024-09-12T09:28:55.581076+0000 (3600 sec) 2024-09-12T08:28:56.379 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:28:56.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/113963503 2024-09-12T08:28:57.133 INFO:tasks.workunit.client.0.smithi137.stderr:listed 27 entries 2024-09-12T08:28:57.168 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:28:57.169 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:29:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.002+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.002+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea092e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.002+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.002+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea160780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.010+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.010+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea160600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.010+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.010+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea160400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.014+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.014+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea160800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.018+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.018+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea160a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.018+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.018+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.022+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.022+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea160e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.026+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.026 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.026+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.030+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.030+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.034+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.034+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.038+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.038+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea160c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.042+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.042+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.046+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.046+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.058+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.058+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.062+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.062+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.066+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.066+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea182000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.070+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.070+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea182200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.074+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.074+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea182400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.078+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:29:00.078+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea182600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:29:07.170 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:29:07.170 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:29:07.170 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:29:07.181 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:29:08.189 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:29:08.190 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:29:08.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129748 2024-09-12T08:29:08.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:29:08.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/113963503 ]] 2024-09-12T08:29:08.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/113963503 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\9\6\3\5\0\3 ]] 2024-09-12T08:29:08.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:29:18.200 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:29:18.201 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:29:18.209 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:29:18.228 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:29:19.175 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:29:19.175 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:29:19.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129759 2024-09-12T08:29:19.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:29:19.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/113963503 ]] 2024-09-12T08:29:19.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/113963503 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\9\6\3\5\0\3 ]] 2024-09-12T08:29:19.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:29:29.179 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:29:29.181 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:29:29.185 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:29:29.193 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:29:30.295 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:29:30.295 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:29:30.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129770 2024-09-12T08:29:30.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:29:30.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/113963503 ]] 2024-09-12T08:29:30.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/113963503 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\9\6\3\5\0\3 ]] 2024-09-12T08:29:30.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:29:40.298 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:29:40.301 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:29:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:29:40.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:29:41.218 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:29:41.218 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:29:41.219 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129781 2024-09-12T08:29:41.219 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:29:41.219 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/113963503 ]] 2024-09-12T08:29:41.219 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/113963503 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\9\6\3\5\0\3 ]] 2024-09-12T08:29:41.219 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:29:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:29:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:29:51.224 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:29:51.232 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:29:52.203 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/113963503 2024-09-12T08:29:52.205 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:29:52.206 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129792 2024-09-12T08:29:52.206 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:29:52.206 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/113963503 ]] 2024-09-12T08:29:52.206 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/113963503 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\9\6\3\5\0\3 ]] 2024-09-12T08:29:52.206 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:29:53.751 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.749+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:29:53.753 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.753+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:29:53.753 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.753+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:29:53.753 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.753+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:29:53.753 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.753+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:29:53.756 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.757+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:29:53.757 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.757+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:29:53.757 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.757+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:29:54.002 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.989+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:29:54.003 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.989+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:29:54.003 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.989+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:29:54.003 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:53.993+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:29:57.253 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:57.252+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:29:57.258 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:57.256+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:29:57.258 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:57.256+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:29:57.258 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:57.256+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:29:57.268 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:29:57.268+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:30:02.208 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:30:02.209 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:30:02.220 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:30:02.227 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:30:03.347 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:30:03.347 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:30:03.348 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129803 2024-09-12T08:30:03.348 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:30:03.348 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:30:03.348 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\1\3\9\6\3\5\0\3 ]] 2024-09-12T08:30:03.348 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/4117469639 2024-09-12T08:30:03.619 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:03.616+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:30:03.627 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:03.624+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:30:03.627 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:03.624+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:30:03.628 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:03.624+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:30:05.347 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/4117469639 until 2024-09-12T09:30:04.697549+0000 (3600 sec) 2024-09-12T08:30:05.391 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:30:05.392 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/4117469639 2024-09-12T08:30:06.212 INFO:tasks.workunit.client.0.smithi137.stderr:listed 28 entries 2024-09-12T08:30:06.237 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:30:06.237 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:30:06.649 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.648+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:30:06.649 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.648+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:30:06.649 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.648+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:30:06.649 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.648+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:30:06.650 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.648+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:30:06.650 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.648+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:30:06.653 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.652+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:30:06.654 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.652+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:30:06.654 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.652+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:30:06.654 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.652+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:30:06.654 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:06.652+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:30:12.134 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:12.124+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:30:12.140 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:12.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:30:12.140 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:12.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:30:12.140 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:12.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:30:12.140 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:12.136+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:30:16.248 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:30:16.250 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:30:16.252 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:30:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:30:17.274 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:30:17.275 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:30:17.275 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129817 2024-09-12T08:30:17.275 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:30:17.275 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:30:17.276 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:30:17.276 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:30:19.265 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:19.264+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:30:19.265 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:19.264+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:30:19.267 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:19.264+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:30:27.278 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:30:27.279 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:30:27.288 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:30:27.297 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:30:28.297 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:30:28.298 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:30:28.301 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129828 2024-09-12T08:30:28.301 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:30:28.301 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:30:28.301 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:30:28.301 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:30:38.305 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:30:38.305 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:30:38.305 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:30:38.305 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:30:39.378 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:30:39.378 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:30:39.379 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129839 2024-09-12T08:30:39.379 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:30:39.379 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:30:39.379 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:30:39.379 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:30:44.775 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:30:44.772+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:30:49.382 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:30:49.386 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:30:49.401 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:30:49.403 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:30:50.382 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:30:50.382 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:30:50.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129850 2024-09-12T08:30:50.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:30:50.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:30:50.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:30:50.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:30:52.641 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:30:52.641 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:30:52.641 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:30:52.641 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:30:52.642 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:30:52.642 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:30:52.642 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:30:52.642 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:30:52.642 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:30:52.642 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:30:52.639+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:31:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cbe500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea182b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.011+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.011+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea182900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.015+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.015+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea182700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea182d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.027+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.027+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea182f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.031+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.031+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.043+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.043+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.092 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.091+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.091+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.095+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.095+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.099+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:31:00.099+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:31:00.220 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:00.219+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:31:00.225 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:00.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:31:00.225 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:00.223+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:31:00.231 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:00.227+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:31:00.388 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:31:00.389 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:31:00.390 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:31:00.392 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:31:01.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:31:01.381 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:31:01.382 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129861 2024-09-12T08:31:01.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:31:01.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:31:01.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:31:01.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:31:11.384 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:31:11.384 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:31:11.385 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:31:11.388 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:31:12.369 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4117469639 2024-09-12T08:31:12.369 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:31:12.370 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129872 2024-09-12T08:31:12.370 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:31:12.370 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4117469639 ]] 2024-09-12T08:31:12.370 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4117469639 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:31:12.370 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:31:15.261 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:31:15.262 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:31:15.262 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:31:15.263 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:31:15.259+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:31:22.375 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:31:22.376 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:31:22.377 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:31:22.379 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:31:23.343 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:31:23.343 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:31:23.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129883 2024-09-12T08:31:23.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:31:23.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1096641937 ]] 2024-09-12T08:31:23.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1096641937 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\1\7\4\6\9\6\3\9 ]] 2024-09-12T08:31:23.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1096641937 2024-09-12T08:31:25.386 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1096641937 until 2024-09-12T09:31:24.603169+0000 (3600 sec) 2024-09-12T08:31:25.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1096641937 2024-09-12T08:31:25.430 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:31:26.195 INFO:tasks.workunit.client.0.smithi137.stderr:listed 29 entries 2024-09-12T08:31:26.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:31:26.230 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:31:36.235 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:31:36.236 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:31:36.236 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:31:36.258 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:31:37.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:31:37.176 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:31:37.177 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129897 2024-09-12T08:31:37.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:31:37.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1096641937 ]] 2024-09-12T08:31:37.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1096641937 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\9\6\6\4\1\9\3\7 ]] 2024-09-12T08:31:37.178 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:31:47.190 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:31:47.191 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:31:47.208 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:31:47.223 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:31:48.202 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:31:48.203 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:31:48.204 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129908 2024-09-12T08:31:48.204 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:31:48.204 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1096641937 ]] 2024-09-12T08:31:48.204 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1096641937 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\9\6\6\4\1\9\3\7 ]] 2024-09-12T08:31:48.204 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:31:58.205 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:31:58.205 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:31:58.216 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:31:58.232 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:31:59.214 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:31:59.214 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:31:59.215 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129919 2024-09-12T08:31:59.215 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:31:59.215 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1096641937 ]] 2024-09-12T08:31:59.215 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1096641937 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\9\6\6\4\1\9\3\7 ]] 2024-09-12T08:31:59.215 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:32:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.006+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea161c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.006+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.010+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.010+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.014+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.014+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.022+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.022+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.042+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.042+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.046+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.046+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.050+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.050+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.054+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.054+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.066+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.066+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.066+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.066+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.066+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.066+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.078+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.078+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.082+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.082+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.086+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.086+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.094+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.094+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.094+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.094+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.098+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.098+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.108 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.102+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.108 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.102+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.112 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.106+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:00.112 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:32:00.106+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:32:09.218 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:32:09.218 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:32:09.220 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:32:09.227 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:32:10.255 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:32:10.255 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:32:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129930 2024-09-12T08:32:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:32:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1096641937 ]] 2024-09-12T08:32:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1096641937 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\9\6\6\4\1\9\3\7 ]] 2024-09-12T08:32:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:32:13.536 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:32:13.534+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:32:13.544 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:32:13.542+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:32:13.545 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:32:13.542+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:32:13.545 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:32:13.542+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:32:13.545 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:32:13.542+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:32:20.258 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:32:20.258 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:32:20.259 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:32:20.269 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:32:21.217 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1096641937 2024-09-12T08:32:21.218 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:32:21.225 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129941 2024-09-12T08:32:21.225 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:32:21.225 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1096641937 ]] 2024-09-12T08:32:21.225 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1096641937 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\9\6\6\4\1\9\3\7 ]] 2024-09-12T08:32:21.225 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:32:31.237 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:32:31.241 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:32:31.242 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:32:31.243 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:32:32.207 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:32:32.207 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:32:32.208 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129952 2024-09-12T08:32:32.208 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:32:32.208 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2006593960 ]] 2024-09-12T08:32:32.208 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2006593960 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\9\6\6\4\1\9\3\7 ]] 2024-09-12T08:32:32.208 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2006593960 2024-09-12T08:32:34.409 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2006593960 until 2024-09-12T09:32:33.656157+0000 (3600 sec) 2024-09-12T08:32:34.449 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:32:34.449 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2006593960 2024-09-12T08:32:35.356 INFO:tasks.workunit.client.0.smithi137.stderr:listed 30 entries 2024-09-12T08:32:35.397 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:32:35.432 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:32:44.268 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:32:44.270+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:32:44.269 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:32:44.270+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:32:45.405 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:32:45.405 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:32:45.407 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:32:45.413 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:32:46.497 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:32:46.497 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:32:46.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129966 2024-09-12T08:32:46.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:32:46.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2006593960 ]] 2024-09-12T08:32:46.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2006593960 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\0\6\5\9\3\9\6\0 ]] 2024-09-12T08:32:46.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:32:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:32:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:32:56.515 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:32:56.516 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:32:57.591 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:32:57.592 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:32:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129977 2024-09-12T08:32:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:32:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2006593960 ]] 2024-09-12T08:32:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2006593960 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\0\6\5\9\3\9\6\0 ]] 2024-09-12T08:32:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:33:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.005+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea14ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.005+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f27980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.013+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.013+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea160d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.017+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.017+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f27680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.021+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.021+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea161d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.033+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.033+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea183600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.041+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.041+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.045+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.045+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea183980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.049+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.049+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eb700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.049+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.051 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.049+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eb280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.053+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.053+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eb080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.057+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.057+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.061+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.061+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eb980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.065+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.065+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0ebb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.069+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.069+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0ebd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.077+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.077+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0ebf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.081+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.081+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea122180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.085+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.085+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea122380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.085+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.085+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea122580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.089+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:33:00.089+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea122780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:33:07.596 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:33:07.597 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:33:07.608 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:33:07.616 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:33:08.547 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:33:08.547 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:33:08.548 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129988 2024-09-12T08:33:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:33:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2006593960 ]] 2024-09-12T08:33:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2006593960 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\0\6\5\9\3\9\6\0 ]] 2024-09-12T08:33:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:33:17.757 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:33:17.753+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:33:17.757 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:33:17.753+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:33:18.550 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:33:18.552 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:33:18.552 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:33:18.566 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:33:19.561 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:33:19.561 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:33:19.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726129999 2024-09-12T08:33:19.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:33:19.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2006593960 ]] 2024-09-12T08:33:19.563 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2006593960 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\0\6\5\9\3\9\6\0 ]] 2024-09-12T08:33:19.563 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:33:29.577 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:33:29.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:33:29.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:33:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:33:30.566 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2006593960 2024-09-12T08:33:30.567 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:33:30.568 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130010 2024-09-12T08:33:30.568 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:33:30.568 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2006593960 ]] 2024-09-12T08:33:30.568 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2006593960 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\0\6\5\9\3\9\6\0 ]] 2024-09-12T08:33:30.568 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:33:40.569 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:33:40.570 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:33:40.580 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:33:40.588 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:33:41.645 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:33:41.647 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:33:41.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130021 2024-09-12T08:33:41.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:33:41.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3728958811 ]] 2024-09-12T08:33:41.649 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3728958811 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\0\0\6\5\9\3\9\6\0 ]] 2024-09-12T08:33:41.649 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3728958811 2024-09-12T08:33:43.419 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3728958811 until 2024-09-12T09:33:42.710745+0000 (3600 sec) 2024-09-12T08:33:43.451 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:33:43.451 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3728958811 2024-09-12T08:33:44.177 INFO:tasks.workunit.client.0.smithi137.stderr:listed 31 entries 2024-09-12T08:33:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:33:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:33:54.226 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:33:54.226 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:33:54.226 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:33:54.240 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:33:55.222 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:33:55.225 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:33:55.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130035 2024-09-12T08:33:55.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:33:55.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3728958811 ]] 2024-09-12T08:33:55.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3728958811 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\7\2\8\9\5\8\8\1\1 ]] 2024-09-12T08:33:55.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:34:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.000+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.000+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d58780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea122f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.008+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.008+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea122d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.020+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.020+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea122b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.028+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea122980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.028+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.044+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.044+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.052+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.052+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.064+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.064+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.076+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.076+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.080+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.080+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.084+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.084+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.096+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.097 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.096+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.104 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.104+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.104 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.104+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.104+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.104+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.108+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.108+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.119 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.116+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.119 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.116+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.121 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.120+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.122 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.120+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.126 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.124+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.126 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.124+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.129 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.128+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:00.129 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:34:00.128+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:34:04.286 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:34:04.280+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:34:05.236 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:34:05.239 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:34:05.245 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:34:05.246 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:34:06.187 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:34:06.191 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:34:06.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130046 2024-09-12T08:34:06.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:34:06.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3728958811 ]] 2024-09-12T08:34:06.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3728958811 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\7\2\8\9\5\8\8\1\1 ]] 2024-09-12T08:34:06.193 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:34:09.765 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:34:09.766 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:34:09.766 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:34:09.766 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:34:09.766 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:34:09.766 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:34:09.766 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:09.760+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:34:16.196 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:34:16.200 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:34:16.200 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:34:16.201 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:34:17.141 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:34:17.142 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:34:17.144 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130057 2024-09-12T08:34:17.144 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:34:17.144 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3728958811 ]] 2024-09-12T08:34:17.144 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3728958811 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\7\2\8\9\5\8\8\1\1 ]] 2024-09-12T08:34:17.144 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:34:27.153 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:34:27.154 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:34:27.155 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:34:27.164 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:34:28.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:34:28.172 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:34:28.172 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130068 2024-09-12T08:34:28.172 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:34:28.172 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3728958811 ]] 2024-09-12T08:34:28.173 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3728958811 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\7\2\8\9\5\8\8\1\1 ]] 2024-09-12T08:34:28.173 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:34:38.174 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:34:38.174 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:34:38.174 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:34:38.180 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:34:39.169 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3728958811 2024-09-12T08:34:39.170 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:34:39.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130079 2024-09-12T08:34:39.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:34:39.170 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3728958811 ]] 2024-09-12T08:34:39.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3728958811 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\7\2\8\9\5\8\8\1\1 ]] 2024-09-12T08:34:39.171 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:34:44.785 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:44.776+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:34:44.786 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:44.784+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:34:44.786 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:44.784+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:34:44.786 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:34:44.784+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:34:49.172 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:34:49.173 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:34:49.173 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:34:49.184 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:34:50.139 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:34:50.139 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:34:50.140 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130090 2024-09-12T08:34:50.140 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:34:50.140 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/558705519 ]] 2024-09-12T08:34:50.140 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/558705519 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\7\2\8\9\5\8\8\1\1 ]] 2024-09-12T08:34:50.140 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/558705519 2024-09-12T08:34:52.406 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/558705519 until 2024-09-12T09:34:51.584465+0000 (3600 sec) 2024-09-12T08:34:52.438 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:34:52.438 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/558705519 2024-09-12T08:34:53.225 INFO:tasks.workunit.client.0.smithi137.stderr:listed 32 entries 2024-09-12T08:34:53.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:34:53.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:35:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f53400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.015+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.015+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9ef3900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d58d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.037 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.101 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.099+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.101 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.099+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.103+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.103+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.107+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.107+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.112 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.111+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.112 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.111+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.126 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.123+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.126 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.123+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.126 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.123+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:00.126 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:35:00.123+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:35:03.247 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:35:03.249 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:35:03.252 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:35:03.262 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:35:04.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:35:04.243 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:35:04.243 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130104 2024-09-12T08:35:04.243 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:35:04.243 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/558705519 ]] 2024-09-12T08:35:04.243 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/558705519 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\5\5\8\7\0\5\5\1\9 ]] 2024-09-12T08:35:04.244 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:35:06.368 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:35:06.369 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:35:06.369 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:35:06.369 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:35:06.369 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:06.363+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:35:14.245 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:35:14.246 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:35:14.256 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:35:14.260 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:35:15.241 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:35:15.241 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:35:15.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130115 2024-09-12T08:35:15.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:35:15.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/558705519 ]] 2024-09-12T08:35:15.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/558705519 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\5\5\8\7\0\5\5\1\9 ]] 2024-09-12T08:35:15.242 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:35:25.244 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:35:25.244 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:35:25.246 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:35:25.246 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:35:26.175 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:35:26.175 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:35:26.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130126 2024-09-12T08:35:26.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:35:26.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/558705519 ]] 2024-09-12T08:35:26.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/558705519 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\5\5\8\7\0\5\5\1\9 ]] 2024-09-12T08:35:26.176 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:35:36.180 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:35:36.181 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:35:36.181 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:35:36.184 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:35:37.248 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:35:37.248 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:35:37.253 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130137 2024-09-12T08:35:37.254 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:35:37.254 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/558705519 ]] 2024-09-12T08:35:37.254 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/558705519 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\5\5\8\7\0\5\5\1\9 ]] 2024-09-12T08:35:37.254 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:35:46.538 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:35:46.535+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:35:46.539 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:35:46.535+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:35:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:35:47.255 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:35:47.259 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:35:47.272 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:35:48.149 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/558705519 2024-09-12T08:35:48.149 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:35:48.150 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130148 2024-09-12T08:35:48.150 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:35:48.150 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/558705519 ]] 2024-09-12T08:35:48.150 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/558705519 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\5\5\8\7\0\5\5\1\9 ]] 2024-09-12T08:35:48.150 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:35:51.608 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:35:51.609 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:35:51.609 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:35:51.609 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:35:51.609 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:35:51.603+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:35:58.152 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:35:58.152 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:35:58.152 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:35:58.152 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:35:59.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:35:59.227 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:35:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130159 2024-09-12T08:35:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:35:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2534148885 ]] 2024-09-12T08:35:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2534148885 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\5\5\8\7\0\5\5\1\9 ]] 2024-09-12T08:35:59.228 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2534148885 2024-09-12T08:36:00.282 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.278+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:36:00.289 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.286+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:36:00.289 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.286+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:36:00.290 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.286+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:36:00.292 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.286+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:36:00.292 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.290+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:36:00.292 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.290+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:36:00.292 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.290+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:36:00.292 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:00.290+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:36:00.480 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.474+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e8b70900 image watch failed: 94042217586688, (107) Transport endpoint is not connected 2024-09-12T08:36:00.480 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.474+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e8b70900 handle_error: handle=94042217586688: (107) Transport endpoint is not connected 2024-09-12T08:36:00.480 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.474+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e9428640 handle_unwatch client blocklisted 2024-09-12T08:36:00.480 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.474+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e8b70900 handle_rewatch: client blocklisted 2024-09-12T08:36:00.480 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.478+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9bfb800 image watch failed: 94042217599360, (107) Transport endpoint is not connected 2024-09-12T08:36:00.481 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.478+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9bfb800 handle_error: handle=94042217599360: (107) Transport endpoint is not connected 2024-09-12T08:36:00.481 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.478+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e9429400 handle_unwatch client blocklisted 2024-09-12T08:36:00.481 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.478+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9bfb800 handle_rewatch: client blocklisted 2024-09-12T08:36:00.484 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e9963800 image watch failed: 94042217591296, (107) Transport endpoint is not connected 2024-09-12T08:36:00.484 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9963800 handle_error: handle=94042217591296: (107) Transport endpoint is not connected 2024-09-12T08:36:00.484 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e9425900 handle_unwatch client blocklisted 2024-09-12T08:36:00.484 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9963800 handle_rewatch: client blocklisted 2024-09-12T08:36:00.485 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9962f00 image watch failed: 94042212407936, (107) Transport endpoint is not connected 2024-09-12T08:36:00.485 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9962f00 handle_error: handle=94042212407936: (107) Transport endpoint is not connected 2024-09-12T08:36:00.485 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e9580320 handle_unwatch client blocklisted 2024-09-12T08:36:00.485 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9962f00 handle_rewatch: client blocklisted 2024-09-12T08:36:00.487 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9bfa000 image watch failed: 94042212392960, (107) Transport endpoint is not connected 2024-09-12T08:36:00.487 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9bfa000 handle_error: handle=94042212392960: (107) Transport endpoint is not connected 2024-09-12T08:36:00.487 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e9580a00 handle_unwatch client blocklisted 2024-09-12T08:36:00.487 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.482+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9bfa000 handle_rewatch: client blocklisted 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9bfb500 image watch failed: 94042217597056, (107) Transport endpoint is not connected 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9bfb500 handle_error: handle=94042217597056: (107) Transport endpoint is not connected 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9962600 image watch failed: 94042217595904, (107) Transport endpoint is not connected 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9962600 handle_error: handle=94042217595904: (107) Transport endpoint is not connected 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e8fcecd0 handle_unwatch client blocklisted 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9962600 handle_rewatch: client blocklisted 2024-09-12T08:36:00.571 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e934e230 handle_unwatch client blocklisted 2024-09-12T08:36:00.572 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.566+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9bfb500 handle_rewatch: client blocklisted 2024-09-12T08:36:00.577 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e9012300 image watch failed: 94042217598208, (107) Transport endpoint is not connected 2024-09-12T08:36:00.577 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9012300 handle_error: handle=94042217598208: (107) Transport endpoint is not connected 2024-09-12T08:36:00.578 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e9112600 image watch failed: 94042217600512, (107) Transport endpoint is not connected 2024-09-12T08:36:00.578 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9112600 handle_error: handle=94042217600512: (107) Transport endpoint is not connected 2024-09-12T08:36:00.578 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e9680640 handle_unwatch client blocklisted 2024-09-12T08:36:00.578 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9012300 handle_rewatch: client blocklisted 2024-09-12T08:36:00.578 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e93de4b0 handle_unwatch client blocklisted 2024-09-12T08:36:00.578 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.574+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9112600 handle_rewatch: client blocklisted 2024-09-12T08:36:00.587 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.582+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9bfa900 image watch failed: 94042217594752, (107) Transport endpoint is not connected 2024-09-12T08:36:00.587 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.582+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9bfa900 handle_error: handle=94042217594752: (107) Transport endpoint is not connected 2024-09-12T08:36:00.587 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.582+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e934f5e0 handle_unwatch client blocklisted 2024-09-12T08:36:00.587 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:00.582+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9bfa900 handle_rewatch: client blocklisted 2024-09-12T08:36:01.459 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2534148885 until 2024-09-12T09:36:00.590346+0000 (3600 sec) 2024-09-12T08:36:01.473 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.473 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.473 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.473 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904f5f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e406adc0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e60372b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587ea0a2a00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e6036dd0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587ea0a2b40 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904fad0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.474 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8d95b80 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.475 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904fd40 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.475 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e406a3c0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.475 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904fc70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.475 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587ea0a3400 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.502 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.502 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.502 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.502 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587ea0a2c80 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea386200 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587ea0a2f00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587ea0a3180 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8d957c0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8da0b40 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9b32500 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea3864c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587e9e74340 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea126b40 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea372d00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.503 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.470+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea387000 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e9923c80 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e94f1d40 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904fe10 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e406b180 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904e5b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8de5cc0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e97296e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e97f2480 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e96bed80 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.506 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e90bbf20 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904ed00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e9b32140 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.498+0000 7f9d2631d640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587ea0a2c80 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d2631d640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587e9078e80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904e4e0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e406b400 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587ea0a3400 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea2b4840 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8de5cc0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.507 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea387100 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.508 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e953b140 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.508 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e98cde00 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.508 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.502+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e955d980 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.508 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.506+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.508 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.506+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.511 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.506+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.511 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.506+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.511 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2534148885 2024-09-12T08:36:01.514 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.510+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.514 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.510+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:36:01.518 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.514+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.518 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.514+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.521 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.518+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.522 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.518+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e8916200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.522 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.518+0000 7f9d2631d640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.523 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.518+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587ea0a3180 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.523 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.518+0000 7f9d2631d640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea386540 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.526 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.522+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.526 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.522+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea0a4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.530 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.526+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.530 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.526+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea183f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.536 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.534+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.537 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.534+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea14fc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.541 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.538+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.541 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.538+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.545 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.542+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.546 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.542+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea0eab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:01.569 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:36:01.566+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e91871a0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:36:02.248 INFO:tasks.workunit.client.0.smithi137.stderr:listed 33 entries 2024-09-12T08:36:02.280 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:36:02.280 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:36:12.284 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:36:12.284 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:36:12.284 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:36:12.284 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:36:13.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:36:13.440 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:36:13.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130173 2024-09-12T08:36:13.441 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:36:13.441 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2534148885 ]] 2024-09-12T08:36:13.441 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2534148885 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\3\4\1\4\8\8\8\5 ]] 2024-09-12T08:36:13.441 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:36:18.834 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:18.830+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:36:18.834 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:18.830+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:36:18.834 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:36:18.830+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:36:23.443 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:36:23.443 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:36:23.447 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:36:23.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:36:24.374 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:36:24.375 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:36:24.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130184 2024-09-12T08:36:24.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:36:24.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2534148885 ]] 2024-09-12T08:36:24.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2534148885 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\3\4\1\4\8\8\8\5 ]] 2024-09-12T08:36:24.376 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:36:34.378 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:36:34.379 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:36:34.382 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:36:34.383 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:36:35.298 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:36:35.299 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:36:35.299 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130195 2024-09-12T08:36:35.300 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:36:35.300 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2534148885 ]] 2024-09-12T08:36:35.300 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2534148885 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\3\4\1\4\8\8\8\5 ]] 2024-09-12T08:36:35.300 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:36:45.301 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:36:45.303 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:36:45.304 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:36:45.309 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:36:46.308 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:36:46.343 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:36:46.343 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130206 2024-09-12T08:36:46.343 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:36:46.343 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2534148885 ]] 2024-09-12T08:36:46.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2534148885 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\3\4\1\4\8\8\8\5 ]] 2024-09-12T08:36:46.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:36:56.317 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:36:56.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:36:56.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:36:56.324 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:36:57.345 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2534148885 2024-09-12T08:36:57.346 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:36:57.346 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130217 2024-09-12T08:36:57.347 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:36:57.347 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2534148885 ]] 2024-09-12T08:36:57.347 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2534148885 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\3\4\1\4\8\8\8\5 ]] 2024-09-12T08:36:57.347 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:37:01.924 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:37:01.925+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:37:01.925 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:37:01.925+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:37:04.302 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:04.301+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:37:04.304 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:04.301+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:37:04.304 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:04.301+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:37:04.306 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:04.305+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:37:07.348 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:37:07.348 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:37:07.351 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:37:07.360 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:37:07.430 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.429+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:37:07.431 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.429+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:37:07.439 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:37:07.440 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:37:07.440 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:07.437+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:37:08.466 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:37:08.466 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:37:08.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130228 2024-09-12T08:37:08.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:37:08.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3456266723 ]] 2024-09-12T08:37:08.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3456266723 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\3\4\1\4\8\8\8\5 ]] 2024-09-12T08:37:08.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3456266723 2024-09-12T08:37:10.473 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3456266723 until 2024-09-12T09:37:09.668290+0000 (3600 sec) 2024-09-12T08:37:10.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:37:10.498 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3456266723 2024-09-12T08:37:11.249 INFO:tasks.workunit.client.0.smithi137.stderr:listed 34 entries 2024-09-12T08:37:11.283 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:37:11.283 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:37:21.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:37:21.303 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:37:21.303 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:37:21.316 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:37:22.380 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:37:22.381 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:37:22.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130242 2024-09-12T08:37:22.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:37:22.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3456266723 ]] 2024-09-12T08:37:22.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3456266723 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\5\6\2\6\6\7\2\3 ]] 2024-09-12T08:37:22.382 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:37:32.389 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:37:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:37:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:37:32.404 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:37:33.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:37:33.344 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:37:33.345 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130253 2024-09-12T08:37:33.345 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:37:33.345 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3456266723 ]] 2024-09-12T08:37:33.345 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3456266723 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\5\6\2\6\6\7\2\3 ]] 2024-09-12T08:37:33.345 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:37:34.685 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:37:34.685+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:37:34.685 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:37:34.685+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:37:43.347 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:37:43.349 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:37:43.361 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:37:43.363 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:37:44.479 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:37:44.479 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:37:44.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130264 2024-09-12T08:37:44.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:37:44.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3456266723 ]] 2024-09-12T08:37:44.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3456266723 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\5\6\2\6\6\7\2\3 ]] 2024-09-12T08:37:44.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:37:52.183 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:37:52.184 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:37:52.185 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:37:52.185 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:37:52.181+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:37:54.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:37:54.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:37:54.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:37:54.485 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:37:55.523 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:37:55.523 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:37:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130275 2024-09-12T08:37:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:37:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3456266723 ]] 2024-09-12T08:37:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3456266723 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\5\6\2\6\6\7\2\3 ]] 2024-09-12T08:37:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:38:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.005+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.005+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.013+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.013+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.021+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.021+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.033+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.033+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.045+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.045+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.069+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.069+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.069+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.069+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8916f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.073+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.073+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.073+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.073+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.077+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.077+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.081+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.081+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.085+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.085+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.089+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.089+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.093+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.093+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.097+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.097+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.097+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.097+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.101+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.101+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.105+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.108 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:00.105+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:38:00.448 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:38:00.449+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:38:05.528 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:38:05.529 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:38:05.532 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:38:05.535 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:38:06.475 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3456266723 2024-09-12T08:38:06.475 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:38:06.476 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130286 2024-09-12T08:38:06.477 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:38:06.477 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3456266723 ]] 2024-09-12T08:38:06.477 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3456266723 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\5\6\2\6\6\7\2\3 ]] 2024-09-12T08:38:06.477 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:38:09.996 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.992+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.996+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.996+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.996+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:38:09.997 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.996+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:38:09.998 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:38:09.996+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:38:16.478 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:38:16.480 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:38:16.483 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:38:16.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:38:17.354 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:38:17.354 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:38:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130297 2024-09-12T08:38:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:38:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/720494988 ]] 2024-09-12T08:38:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/720494988 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\5\6\2\6\6\7\2\3 ]] 2024-09-12T08:38:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/720494988 2024-09-12T08:38:19.462 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/720494988 until 2024-09-12T09:38:18.704760+0000 (3600 sec) 2024-09-12T08:38:19.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:38:19.517 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/720494988 2024-09-12T08:38:20.342 INFO:tasks.workunit.client.0.smithi137.stderr:listed 35 entries 2024-09-12T08:38:20.366 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:38:20.366 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:38:30.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:38:30.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:38:30.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:38:30.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:38:31.430 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:38:31.430 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:38:31.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130311 2024-09-12T08:38:31.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:38:31.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/720494988 ]] 2024-09-12T08:38:31.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/720494988 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\2\0\4\9\4\9\8\8 ]] 2024-09-12T08:38:31.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:38:41.437 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:38:41.438 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:38:41.438 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:38:41.448 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:38:42.407 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:38:42.408 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:38:42.409 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130322 2024-09-12T08:38:42.409 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:38:42.409 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/720494988 ]] 2024-09-12T08:38:42.409 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/720494988 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\2\0\4\9\4\9\8\8 ]] 2024-09-12T08:38:42.409 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:38:52.411 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:38:52.417 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:38:52.442 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:38:52.444 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:38:53.374 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:38:53.375 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:38:53.375 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130333 2024-09-12T08:38:53.375 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:38:53.375 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/720494988 ]] 2024-09-12T08:38:53.375 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/720494988 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\2\0\4\9\4\9\8\8 ]] 2024-09-12T08:38:53.375 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:39:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:59.999+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:38:59.999+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cbfa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9f27980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.011+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.011+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.015+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.015+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.027+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.027+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.031+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.031+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31c080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.051+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.051+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31c680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31c880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31ca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:39:03.382 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:39:03.384 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:39:03.389 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:39:03.407 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:39:04.471 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:39:04.472 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:39:04.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130344 2024-09-12T08:39:04.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:39:04.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/720494988 ]] 2024-09-12T08:39:04.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/720494988 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\2\0\4\9\4\9\8\8 ]] 2024-09-12T08:39:04.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:39:14.475 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:39:14.479 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:39:14.479 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:39:14.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:39:15.546 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/720494988 2024-09-12T08:39:15.546 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:39:15.547 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130355 2024-09-12T08:39:15.547 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:39:15.547 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/720494988 ]] 2024-09-12T08:39:15.547 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/720494988 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\2\0\4\9\4\9\8\8 ]] 2024-09-12T08:39:15.547 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:39:25.550 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:39:25.550 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:39:25.551 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:39:25.551 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:39:26.589 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:39:26.589 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:39:26.590 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130366 2024-09-12T08:39:26.590 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:39:26.590 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4054511077 ]] 2024-09-12T08:39:26.590 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4054511077 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\7\2\0\4\9\4\9\8\8 ]] 2024-09-12T08:39:26.591 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/4054511077 2024-09-12T08:39:28.487 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/4054511077 until 2024-09-12T09:39:27.726109+0000 (3600 sec) 2024-09-12T08:39:28.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:39:28.560 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/4054511077 2024-09-12T08:39:29.348 INFO:tasks.workunit.client.0.smithi137.stderr:listed 36 entries 2024-09-12T08:39:29.378 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:39:29.378 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:39:38.747 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:39:38.743+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:39:38.747 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:39:38.743+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:39:39.385 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:39:39.386 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:39:39.389 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:39:39.396 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:39:40.425 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:39:40.425 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:39:40.427 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130380 2024-09-12T08:39:40.427 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:39:40.427 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4054511077 ]] 2024-09-12T08:39:40.427 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4054511077 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\4\5\1\1\0\7\7 ]] 2024-09-12T08:39:40.427 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:39:50.429 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:39:50.430 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:39:50.434 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:39:50.436 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:39:51.510 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:39:51.511 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:39:51.512 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130391 2024-09-12T08:39:51.512 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:39:51.512 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4054511077 ]] 2024-09-12T08:39:51.512 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4054511077 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\4\5\1\1\0\7\7 ]] 2024-09-12T08:39:51.513 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:40:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:59.998+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:39:59.998+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.002+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.002+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.006+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.006+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.010+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.010+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.030+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.030+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.034+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.034+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.042+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.042+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.046+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.046+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.054+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.054+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.054+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.054+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.058+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.058+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.062+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.062+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e94d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.066+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.066+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9e95900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.070+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.073 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.070+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.074+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.074+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9e95d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.078+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.078+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9e95f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.082+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.082+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea328100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.086+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.086+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea328300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.098+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.098+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea328500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.104 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.102+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:00.105 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:40:00.102+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea328700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:40:01.514 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:40:01.515 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:40:01.516 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:40:01.532 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:40:02.572 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:40:02.572 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:40:02.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130402 2024-09-12T08:40:02.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:40:02.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4054511077 ]] 2024-09-12T08:40:02.573 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4054511077 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\4\5\1\1\0\7\7 ]] 2024-09-12T08:40:02.574 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:40:03.459 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:40:03.458+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:40:03.459 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:40:03.458+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:40:12.583 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:40:12.587 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:40:12.593 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:40:12.596 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:40:13.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:40:13.515 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:40:13.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130413 2024-09-12T08:40:13.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:40:13.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4054511077 ]] 2024-09-12T08:40:13.516 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4054511077 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\4\5\1\1\0\7\7 ]] 2024-09-12T08:40:13.517 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:40:15.004 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:40:15.002+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:40:15.026 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:40:15.018+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:40:22.867 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:40:22.866+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:40:22.867 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:40:22.866+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:40:23.521 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:40:23.521 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:40:23.521 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:40:23.526 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:40:24.720 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4054511077 2024-09-12T08:40:24.720 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:40:24.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130424 2024-09-12T08:40:24.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:40:24.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4054511077 ]] 2024-09-12T08:40:24.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4054511077 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\4\5\1\1\0\7\7 ]] 2024-09-12T08:40:24.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:40:34.728 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:40:34.728 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:40:34.744 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:40:34.753 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:40:35.764 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:40:35.765 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:40:35.766 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130435 2024-09-12T08:40:35.766 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:40:35.766 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2966137166 ]] 2024-09-12T08:40:35.766 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2966137166 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\0\5\4\5\1\1\0\7\7 ]] 2024-09-12T08:40:35.766 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2966137166 2024-09-12T08:40:36.213 INFO:tasks.workunit.client.0.smithi137.stderr:2024-09-12T08:40:36.213+0000 7efe996ec640 0 --2- 172.21.15.137:0/1466316461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe80003a60 0x7efe80003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-12T08:40:38.536 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2966137166 until 2024-09-12T09:40:37.788289+0000 (3600 sec) 2024-09-12T08:40:38.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:40:38.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2966137166 2024-09-12T08:40:39.407 INFO:tasks.workunit.client.0.smithi137.stderr:listed 37 entries 2024-09-12T08:40:39.439 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:40:39.439 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:40:49.441 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:40:49.441 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:40:49.442 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:40:49.442 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:40:50.567 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:40:50.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:40:50.576 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130450 2024-09-12T08:40:50.576 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:40:50.576 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2966137166 ]] 2024-09-12T08:40:50.576 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2966137166 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\6\6\1\3\7\1\6\6 ]] 2024-09-12T08:40:50.576 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:41:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.001+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.001+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.005+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.005+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea0eb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.005+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.005+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9cfae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.009+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.009+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea0eb900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.013+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.013+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea161a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.017+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.017+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.017+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.017+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.021+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.021+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.025+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.025+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31d700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.025+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.025+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.041+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.041+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.053+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.053+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.057+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.057+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea14e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.061+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.061+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9e94c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.065+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.065+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea31d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.069+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.069+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e8917700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.073+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.073+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f26780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.077+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.077+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea0a5d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.093+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.093+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea175700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.097+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:41:00.097+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:41:00.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:41:00.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:41:00.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:41:00.580 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:41:01.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:41:01.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:41:01.579 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130461 2024-09-12T08:41:01.579 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:41:01.579 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2966137166 ]] 2024-09-12T08:41:01.579 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2966137166 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\6\6\1\3\7\1\6\6 ]] 2024-09-12T08:41:01.579 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:41:04.766 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:41:04.765+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:41:04.767 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:41:04.765+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:41:11.582 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:41:11.582 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:41:11.582 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:41:11.588 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:41:12.478 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:41:12.479 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:41:12.479 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130472 2024-09-12T08:41:12.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:41:12.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2966137166 ]] 2024-09-12T08:41:12.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2966137166 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\6\6\1\3\7\1\6\6 ]] 2024-09-12T08:41:12.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:41:22.488 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:41:22.488 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:41:22.488 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:41:22.497 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:41:23.434 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:41:23.435 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:41:23.435 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130483 2024-09-12T08:41:23.435 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:41:23.435 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2966137166 ]] 2024-09-12T08:41:23.435 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2966137166 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\6\6\1\3\7\1\6\6 ]] 2024-09-12T08:41:23.436 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:41:24.216 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:41:24.213+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:41:33.438 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:41:33.438 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:41:33.443 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:41:33.443 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:41:34.460 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2966137166 2024-09-12T08:41:34.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:41:34.461 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130494 2024-09-12T08:41:34.461 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:41:34.461 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2966137166 ]] 2024-09-12T08:41:34.461 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2966137166 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\6\6\1\3\7\1\6\6 ]] 2024-09-12T08:41:34.461 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:41:44.465 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:41:44.466 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:41:44.475 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:41:44.476 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:41:45.474 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:41:45.480 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:41:45.481 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130505 2024-09-12T08:41:45.481 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:41:45.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1669362779 ]] 2024-09-12T08:41:45.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1669362779 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\9\6\6\1\3\7\1\6\6 ]] 2024-09-12T08:41:45.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1669362779 2024-09-12T08:41:47.537 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1669362779 until 2024-09-12T09:41:46.713887+0000 (3600 sec) 2024-09-12T08:41:47.565 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:41:47.565 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1669362779 2024-09-12T08:41:48.381 INFO:tasks.workunit.client.0.smithi137.stderr:listed 38 entries 2024-09-12T08:41:48.416 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:41:48.416 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:41:48.701 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:41:48.702 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:41:48.700+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:41:58.430 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:41:58.430 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:41:58.432 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:41:58.436 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:41:59.418 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:41:59.427 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:41:59.428 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130519 2024-09-12T08:41:59.428 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:41:59.428 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1669362779 ]] 2024-09-12T08:41:59.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1669362779 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\9\3\6\2\7\7\9 ]] 2024-09-12T08:41:59.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:42:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.008+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.008+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e886cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.008+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.008+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea093000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.012+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.012+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a5c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.016+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.016+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea093880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.020+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.020+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea14e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.024+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.024+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea161f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.028+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.032+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea183f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.040+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.040+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0ebc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.048+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.048+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8916180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.052+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.052+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8917500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.060+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.060+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8916c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.068+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.068+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eb880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.072+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.072+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea123380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.076+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.076+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8917200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.080+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.080+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8917000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.084+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.084+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea175580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.088+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.088+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea174b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.096+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.096+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea175680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.106 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.104+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.106 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.104+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea328800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.108+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:42:00.108+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea328980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:42:09.432 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:42:09.435 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:42:09.435 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:42:09.440 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:42:09.578 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:42:09.579 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:42:09.579 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:42:09.579 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:09.576+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:42:10.419 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:42:10.420 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:42:10.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130530 2024-09-12T08:42:10.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:42:10.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1669362779 ]] 2024-09-12T08:42:10.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1669362779 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\9\3\6\2\7\7\9 ]] 2024-09-12T08:42:10.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:42:20.424 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:42:20.429 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:42:20.437 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:42:20.442 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:42:21.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:42:21.467 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:42:21.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130541 2024-09-12T08:42:21.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:42:21.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1669362779 ]] 2024-09-12T08:42:21.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1669362779 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\9\3\6\2\7\7\9 ]] 2024-09-12T08:42:21.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:42:23.285 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:42:23.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.283+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:42:23.962 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:42:23.963 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:42:23.963 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:42:23.963 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:23.959+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:42:30.943 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.939+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:42:30.947 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.943+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:42:30.957 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.947+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:42:30.957 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.947+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:42:30.957 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.947+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.947+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.947+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.951+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.951+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.951+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.951+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:42:30.958 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:42:30.951+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:42:31.469 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:42:31.470 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:42:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:42:31.485 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:42:32.453 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:42:32.453 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:42:32.453 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130552 2024-09-12T08:42:32.454 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:42:32.454 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1669362779 ]] 2024-09-12T08:42:32.454 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1669362779 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\9\3\6\2\7\7\9 ]] 2024-09-12T08:42:32.454 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:42:42.455 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:42:42.457 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:42:42.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:42:42.468 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:42:43.380 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1669362779 2024-09-12T08:42:43.380 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:42:43.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130563 2024-09-12T08:42:43.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:42:43.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1669362779 ]] 2024-09-12T08:42:43.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1669362779 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\9\3\6\2\7\7\9 ]] 2024-09-12T08:42:43.381 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:42:53.385 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:42:53.387 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:42:53.396 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:42:53.412 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:42:54.089 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:42:54.083+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:42:54.089 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:42:54.083+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:42:54.395 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:42:54.395 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:42:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130574 2024-09-12T08:42:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:42:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3407300315 ]] 2024-09-12T08:42:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3407300315 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\6\6\9\3\6\2\7\7\9 ]] 2024-09-12T08:42:54.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3407300315 2024-09-12T08:42:56.582 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3407300315 until 2024-09-12T09:42:55.822837+0000 (3600 sec) 2024-09-12T08:42:56.636 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:42:56.636 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3407300315 2024-09-12T08:42:57.402 INFO:tasks.workunit.client.0.smithi137.stderr:listed 39 entries 2024-09-12T08:42:57.437 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:42:57.437 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:43:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea183f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.003+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea328680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.007+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea328580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.011+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.011+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea328e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.019+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.019+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.027+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.027+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.039+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.039+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.043+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.043+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.055+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.055+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.055+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea329c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.059+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.059+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea329500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:43:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:43:01.157 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:01.155+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:43:07.439 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:43:07.439 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:43:07.450 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:43:07.458 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:43:08.371 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:43:08.371 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:43:08.372 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130588 2024-09-12T08:43:08.372 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:43:08.372 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3407300315 ]] 2024-09-12T08:43:08.372 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3407300315 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\0\7\3\0\0\3\1\5 ]] 2024-09-12T08:43:08.372 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:43:18.374 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:43:18.374 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:43:18.374 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:43:18.379 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:43:19.397 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:43:19.397 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:43:19.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130599 2024-09-12T08:43:19.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:43:19.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3407300315 ]] 2024-09-12T08:43:19.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3407300315 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\0\7\3\0\0\3\1\5 ]] 2024-09-12T08:43:19.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:43:29.402 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:43:29.402 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:43:29.402 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:43:29.421 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:43:30.428 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:43:30.428 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:43:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130610 2024-09-12T08:43:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:43:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3407300315 ]] 2024-09-12T08:43:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3407300315 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\0\7\3\0\0\3\1\5 ]] 2024-09-12T08:43:30.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:43:40.430 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:43:40.431 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:43:40.433 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:43:40.435 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:43:41.367 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:43:41.367 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:43:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130621 2024-09-12T08:43:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:43:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3407300315 ]] 2024-09-12T08:43:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3407300315 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\0\7\3\0\0\3\1\5 ]] 2024-09-12T08:43:41.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:43:50.882 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.878+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:43:50.896 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.894+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:43:50.896 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.894+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:43:50.896 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.894+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:43:50.896 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.894+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:43:50.896 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.894+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:43:50.901 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.898+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:43:50.901 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.898+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:43:50.901 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.898+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:43:50.901 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.898+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:43:50.901 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:43:50.898+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:43:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:43:51.370 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:43:51.377 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:43:51.386 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:43:52.330 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3407300315 2024-09-12T08:43:52.330 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:43:52.331 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130632 2024-09-12T08:43:52.331 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:43:52.331 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3407300315 ]] 2024-09-12T08:43:52.331 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3407300315 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\0\7\3\0\0\3\1\5 ]] 2024-09-12T08:43:52.331 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:44:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:44:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:44:02.333 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:44:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:44:03.338 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:44:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:44:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130643 2024-09-12T08:44:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:44:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2446617153 ]] 2024-09-12T08:44:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2446617153 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\4\0\7\3\0\0\3\1\5 ]] 2024-09-12T08:44:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2446617153 2024-09-12T08:44:05.594 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2446617153 until 2024-09-12T09:44:04.812295+0000 (3600 sec) 2024-09-12T08:44:05.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:44:05.637 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2446617153 2024-09-12T08:44:05.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:44:05.953+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:44:05.967 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:44:05.953+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:44:05.967 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:44:05.961+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:44:06.497 INFO:tasks.workunit.client.0.smithi137.stderr:listed 40 entries 2024-09-12T08:44:06.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:44:06.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:44:14.585 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:44:14.585+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:44:16.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:44:16.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:44:16.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:44:16.533 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:44:17.525 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:44:17.533 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:44:17.537 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130657 2024-09-12T08:44:17.537 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:44:17.537 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2446617153 ]] 2024-09-12T08:44:17.537 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2446617153 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\4\4\6\6\1\7\1\5\3 ]] 2024-09-12T08:44:17.538 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:44:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:44:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:44:27.544 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:44:27.562 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:44:28.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:44:28.630 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:44:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130668 2024-09-12T08:44:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:44:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2446617153 ]] 2024-09-12T08:44:28.632 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2446617153 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\4\4\6\6\1\7\1\5\3 ]] 2024-09-12T08:44:28.632 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:44:38.633 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:44:38.634 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:44:38.634 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:44:38.637 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:44:39.557 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:44:39.557 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:44:39.558 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130679 2024-09-12T08:44:39.558 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:44:39.558 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2446617153 ]] 2024-09-12T08:44:39.558 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2446617153 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\4\4\6\6\1\7\1\5\3 ]] 2024-09-12T08:44:39.558 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:44:49.561 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:44:49.561 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:44:49.561 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:44:49.570 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:44:50.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:44:50.654 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:44:50.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130690 2024-09-12T08:44:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:44:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2446617153 ]] 2024-09-12T08:44:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2446617153 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\4\4\6\6\1\7\1\5\3 ]] 2024-09-12T08:44:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:45:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.000+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.000+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.000+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.000+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.004+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.004+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.008+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.008+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9616d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.012+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.012+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.016+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.016+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.020+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.020+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.052+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.052+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.056+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.056+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9e95a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.060+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.060+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.068+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.068+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea31db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.072+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.072+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.076+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.076+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.080+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.080+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.084+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.084+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.088+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.088+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.092+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.092+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.096+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.096+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.100+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:00.100+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:45:00.657 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:45:00.661 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:45:00.663 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:45:00.666 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:45:01.627 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2446617153 2024-09-12T08:45:01.627 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:45:01.628 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130701 2024-09-12T08:45:01.628 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:45:01.628 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2446617153 ]] 2024-09-12T08:45:01.628 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2446617153 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\4\4\6\6\1\7\1\5\3 ]] 2024-09-12T08:45:01.629 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:45:11.632 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:45:11.632 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:45:11.633 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:45:11.656 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:45:12.697 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:45:12.700 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:45:12.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130712 2024-09-12T08:45:12.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:45:12.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:45:12.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\4\4\6\6\1\7\1\5\3 ]] 2024-09-12T08:45:12.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/965763560 2024-09-12T08:45:14.582 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/965763560 until 2024-09-12T09:45:13.857819+0000 (3600 sec) 2024-09-12T08:45:14.632 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:45:14.637 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/965763560 2024-09-12T08:45:15.050 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:15.048+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:45:15.050 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:15.048+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:45:15.055 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:15.052+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:45:15.501 INFO:tasks.workunit.client.0.smithi137.stderr:listed 41 entries 2024-09-12T08:45:15.529 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:45:15.530 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:45:25.551 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:45:25.551 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:45:25.560 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:45:25.578 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:45:26.596 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:45:26.623 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:45:26.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130726 2024-09-12T08:45:26.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:45:26.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:45:26.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:45:26.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:45:36.626 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:45:36.629 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:45:36.630 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:45:36.630 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:45:36.875 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.864+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:45:36.878 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.876+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:45:36.879 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.876+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:45:36.884 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.880+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:45:36.884 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.880+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:45:36.898 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.896+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:45:36.899 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.896+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:45:36.899 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:45:36.896+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:45:37.574 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:45:37.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:45:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130737 2024-09-12T08:45:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:45:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:45:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:45:37.575 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:45:44.783 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:45:44.779+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:45:44.783 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:45:44.779+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:45:46.312 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:45:46.307+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:45:46.312 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:45:46.307+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:45:47.590 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:45:47.590 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:45:47.597 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:45:47.607 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:45:48.490 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:45:48.490 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:45:48.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130748 2024-09-12T08:45:48.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:45:48.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:45:48.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:45:48.491 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:45:58.502 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:45:58.502 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:45:58.504 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:45:58.521 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:45:59.444 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:45:59.444 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:45:59.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130759 2024-09-12T08:45:59.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:45:59.445 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:45:59.446 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:45:59.446 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:46:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:59.999+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:45:59.999+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.011+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.011+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.019+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.019+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.023+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.031+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.031+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.040 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.043+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.043+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.051+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.051+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.071 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.091+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.091+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.095+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.095+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.099+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:46:00.099+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:46:09.452 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:46:09.452 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:46:09.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:46:09.472 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:46:10.436 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:46:10.436 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:46:10.437 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130770 2024-09-12T08:46:10.437 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:46:10.437 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:46:10.437 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:46:10.438 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:46:19.588 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.575+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:46:19.588 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.579+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:46:19.588 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.579+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:46:19.589 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.579+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:46:19.589 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.579+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:46:19.589 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.583+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:46:19.592 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.583+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:46:19.593 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.583+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:46:19.593 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.583+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:46:19.593 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:19.583+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:46:20.443 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:46:20.444 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:46:20.456 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:46:20.457 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:46:21.488 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/965763560 2024-09-12T08:46:21.488 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:46:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130781 2024-09-12T08:46:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:46:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/965763560 ]] 2024-09-12T08:46:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/965763560 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:46:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:46:31.495 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:46:31.496 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:46:31.497 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:46:31.511 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:46:32.471 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:46:32.471 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:46:32.472 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130792 2024-09-12T08:46:32.472 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:46:32.472 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2613560026 ]] 2024-09-12T08:46:32.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2613560026 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\9\6\5\7\6\3\5\6\0 ]] 2024-09-12T08:46:32.473 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2613560026 2024-09-12T08:46:34.592 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2613560026 until 2024-09-12T09:46:33.798433+0000 (3600 sec) 2024-09-12T08:46:34.623 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:46:34.631 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2613560026 2024-09-12T08:46:35.368 INFO:tasks.workunit.client.0.smithi137.stderr:listed 42 entries 2024-09-12T08:46:35.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:46:35.431 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:46:45.433 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:46:45.433 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:46:45.447 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:46:45.449 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:46:46.361 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:46:46.366 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:46:46.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130806 2024-09-12T08:46:46.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:46:46.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2613560026 ]] 2024-09-12T08:46:46.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2613560026 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\6\1\3\5\6\0\0\2\6 ]] 2024-09-12T08:46:46.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:46:49.223 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:46:49.218+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:46:49.223 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:46:49.218+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:46:54.319 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:54.310+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:46:54.319 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:54.310+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:46:54.328 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:46:54.322+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:46:56.374 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:46:56.374 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:46:56.378 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:46:56.391 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:46:57.284 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:46:57.290 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:46:57.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130817 2024-09-12T08:46:57.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:46:57.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2613560026 ]] 2024-09-12T08:46:57.292 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2613560026 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\6\1\3\5\6\0\0\2\6 ]] 2024-09-12T08:46:57.292 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:47:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.006+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.006+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f26680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.010+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.010+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea329e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.014+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.014+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea174f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.017 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.018+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.018+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea31c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.018+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.018+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea174980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.042+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.042+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.054+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.054+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.058+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.058 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.058+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b87480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.062+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.062+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0ea680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.066+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.066+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8916980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.070+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.070+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8917100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.074+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.074+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.078+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.078+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.082+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.082+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.102+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.102+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.105 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.102+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.105 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.102+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.106+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.106+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.111 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.110+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.111 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.110+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.115 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.114+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.115 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.114+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.118+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:00.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:47:00.118+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:47:07.295 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:47:07.295 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:47:07.296 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:47:07.305 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:47:08.419 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:47:08.419 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:47:08.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130828 2024-09-12T08:47:08.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:47:08.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2613560026 ]] 2024-09-12T08:47:08.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2613560026 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\6\1\3\5\6\0\0\2\6 ]] 2024-09-12T08:47:08.420 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:47:18.441 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:47:18.442 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:47:18.443 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:47:18.445 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:47:19.462 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:47:19.462 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:47:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130839 2024-09-12T08:47:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:47:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2613560026 ]] 2024-09-12T08:47:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2613560026 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\6\1\3\5\6\0\0\2\6 ]] 2024-09-12T08:47:19.463 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:47:20.419 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:47:20.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:47:20.421 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:20.418+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.338+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:47:22.346 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:47:22.347 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:47:22.347 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.342+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:47:22.669 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:22.666+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:29.466 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:47:29.466 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:47:29.482 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:47:29.490 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:47:30.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2613560026 2024-09-12T08:47:30.481 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:47:30.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130850 2024-09-12T08:47:30.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:47:30.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2613560026 ]] 2024-09-12T08:47:30.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2613560026 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\6\1\3\5\6\0\0\2\6 ]] 2024-09-12T08:47:30.483 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:47:38.547 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:38.545+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:47:38.548 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:38.545+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:47:38.548 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:38.545+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:47:38.549 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:38.545+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:40.499 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:47:40.503 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:47:40.507 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:47:40.517 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:47:40.916 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.913+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:47:40.920 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:47:40.921 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:40.917+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:47:41.467 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:47:41.467 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:47:41.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130861 2024-09-12T08:47:41.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:47:41.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2267109441 ]] 2024-09-12T08:47:41.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2267109441 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\6\1\3\5\6\0\0\2\6 ]] 2024-09-12T08:47:41.468 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2267109441 2024-09-12T08:47:43.578 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2267109441 until 2024-09-12T09:47:42.848064+0000 (3600 sec) 2024-09-12T08:47:43.616 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2267109441 2024-09-12T08:47:43.617 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:47:44.358 INFO:tasks.workunit.client.0.smithi137.stderr:listed 43 entries 2024-09-12T08:47:44.385 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:47:44.385 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:47:45.591 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:45.589+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:47.290 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:47.285+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:47:47.296 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:47.289+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:47:47.296 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:47.289+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:47:54.393 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:47:54.397 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:47:54.397 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:47:54.406 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:47:55.383 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:47:55.383 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:47:55.384 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130875 2024-09-12T08:47:55.384 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:47:55.384 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2267109441 ]] 2024-09-12T08:47:55.384 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2267109441 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\6\7\1\0\9\4\4\1 ]] 2024-09-12T08:47:55.384 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:47:56.167 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.165+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:47:56.174 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.173+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:47:56.189 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:47:56.190 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:47:56.190 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:47:56.185+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:48:00.003 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.001+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.001+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.005+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.005+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e8916980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.005+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.005+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e8917100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.009+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.009+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e8917000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.013+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.013 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.013+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea14ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.017+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.018 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.017+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9e94900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.021+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.021+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b87c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.021+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.021+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b87700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.037+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.039 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.037+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e6980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.041+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.041+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.045+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.045+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.049+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.049+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.057+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.057+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.061+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.061+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.065+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.065+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.069+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.072 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.069+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.073+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.073+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.085+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.085+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.089+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:48:00.089+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:48:05.386 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:48:05.390 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:48:05.392 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:48:05.403 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:48:05.491 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:48:05.489+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:48:05.491 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:48:05.489+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:48:06.348 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:48:06.349 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:48:06.349 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130886 2024-09-12T08:48:06.349 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:48:06.349 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2267109441 ]] 2024-09-12T08:48:06.349 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2267109441 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\6\7\1\0\9\4\4\1 ]] 2024-09-12T08:48:06.350 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:48:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:48:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:48:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:48:16.352 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:48:17.343 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:48:17.343 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:48:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130897 2024-09-12T08:48:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:48:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2267109441 ]] 2024-09-12T08:48:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2267109441 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\6\7\1\0\9\4\4\1 ]] 2024-09-12T08:48:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:48:27.346 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:48:27.346 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:48:27.346 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:48:27.350 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:48:28.325 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:48:28.325 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:48:28.325 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130908 2024-09-12T08:48:28.326 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:48:28.326 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2267109441 ]] 2024-09-12T08:48:28.326 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2267109441 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\6\7\1\0\9\4\4\1 ]] 2024-09-12T08:48:28.326 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:48:38.328 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:48:38.328 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:48:38.331 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:48:38.346 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:48:39.358 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2267109441 2024-09-12T08:48:39.358 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:48:39.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130919 2024-09-12T08:48:39.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:48:39.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2267109441 ]] 2024-09-12T08:48:39.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2267109441 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\6\7\1\0\9\4\4\1 ]] 2024-09-12T08:48:39.359 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:48:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:48:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:48:49.366 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:48:49.370 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:48:50.439 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:48:50.439 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:48:50.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130930 2024-09-12T08:48:50.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:48:50.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1022528467 ]] 2024-09-12T08:48:50.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1022528467 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\2\6\7\1\0\9\4\4\1 ]] 2024-09-12T08:48:50.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1022528467 2024-09-12T08:48:52.618 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1022528467 until 2024-09-12T09:48:51.854098+0000 (3600 sec) 2024-09-12T08:48:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1022528467 2024-09-12T08:48:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:48:53.437 INFO:tasks.workunit.client.0.smithi137.stderr:listed 44 entries 2024-09-12T08:48:53.474 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:48:53.474 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:49:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.016+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.016+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e899ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.016+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.016+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea329c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.032+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.032+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea329980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.040+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.040+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea329c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.044+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.044+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea329b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.052+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.052+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f26600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.060+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.060+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea174b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.072+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.076 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.072+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea174a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.076+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.076+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea160c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.080+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.084 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.080+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e6f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.084+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.084+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.088+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.088+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea174880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.092+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.092+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.096+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.096+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.103 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.100+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.104 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.100+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9cfaf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.104+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.108+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.118 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.116+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.119 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.116+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.128 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.124+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.128 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.124+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.138 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.136+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.138 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.136+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.144 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.140+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:00.144 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:49:00.140+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:49:03.478 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:49:03.481 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:49:03.485 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:49:03.485 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:49:04.351 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:49:04.351 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:49:04.351 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130944 2024-09-12T08:49:04.351 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:49:04.352 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1022528467 ]] 2024-09-12T08:49:04.352 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1022528467 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\2\2\5\2\8\4\6\7 ]] 2024-09-12T08:49:04.352 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:49:14.362 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:49:14.363 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:49:14.364 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:49:14.366 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:49:15.297 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:49:15.303 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:49:15.304 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130955 2024-09-12T08:49:15.304 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:49:15.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1022528467 ]] 2024-09-12T08:49:15.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1022528467 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\2\2\5\2\8\4\6\7 ]] 2024-09-12T08:49:15.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:49:25.310 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:49:25.311 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:49:25.313 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:49:25.313 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130966 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1022528467 ]] 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1022528467 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\2\2\5\2\8\4\6\7 ]] 2024-09-12T08:49:26.393 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:49:36.406 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:49:36.407 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:49:36.416 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:49:36.418 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:49:37.360 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:49:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:49:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130977 2024-09-12T08:49:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:49:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1022528467 ]] 2024-09-12T08:49:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1022528467 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\2\2\5\2\8\4\6\7 ]] 2024-09-12T08:49:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:49:43.655 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:49:43.635+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:49:43.655 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:49:43.635+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:49:43.655 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:49:43.639+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:49:47.363 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:49:47.367 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:49:47.373 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:49:47.374 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:49:48.311 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1022528467 2024-09-12T08:49:48.311 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:49:48.312 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130988 2024-09-12T08:49:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:49:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1022528467 ]] 2024-09-12T08:49:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1022528467 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\2\2\5\2\8\4\6\7 ]] 2024-09-12T08:49:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:49:58.325 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:49:58.325 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:49:58.325 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:49:58.332 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:49:59.283 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:49:59.283 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:49:59.284 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726130999 2024-09-12T08:49:59.284 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:49:59.284 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4107428414 ]] 2024-09-12T08:49:59.284 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4107428414 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\0\2\2\5\2\8\4\6\7 ]] 2024-09-12T08:49:59.284 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/4107428414 2024-09-12T08:50:00.675 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e956f800 image watch failed: 94042212407936, (107) Transport endpoint is not connected 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e956f800 handle_error: handle=94042212407936: (107) Transport endpoint is not connected 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e88db7c0 handle_unwatch client blocklisted 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e956f800 handle_rewatch: client blocklisted 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9151200 image watch failed: 94042217586688, (107) Transport endpoint is not connected 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9151200 handle_error: handle=94042217586688: (107) Transport endpoint is not connected 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e8cc77c0 handle_unwatch client blocklisted 2024-09-12T08:50:00.676 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9151200 handle_rewatch: client blocklisted 2024-09-12T08:50:00.677 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9012300 image watch failed: 94042212406784, (107) Transport endpoint is not connected 2024-09-12T08:50:00.677 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.671+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9012300 handle_error: handle=94042212406784: (107) Transport endpoint is not connected 2024-09-12T08:50:00.677 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9a52600 image watch failed: 94042217591296, (107) Transport endpoint is not connected 2024-09-12T08:50:00.677 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9a52600 handle_error: handle=94042217591296: (107) Transport endpoint is not connected 2024-09-12T08:50:00.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e9425450 handle_unwatch client blocklisted 2024-09-12T08:50:00.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9012300 handle_rewatch: client blocklisted 2024-09-12T08:50:00.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::watcher::RewatchRequest: 0x5587e8b73360 handle_unwatch client blocklisted 2024-09-12T08:50:00.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9a52600 handle_rewatch: client blocklisted 2024-09-12T08:50:00.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9a53800 image watch failed: 94042212392960, (107) Transport endpoint is not connected 2024-09-12T08:50:00.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9a53800 handle_error: handle=94042212392960: (107) Transport endpoint is not connected 2024-09-12T08:50:00.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e95d7f90 handle_unwatch client blocklisted 2024-09-12T08:50:00.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.675+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9a53800 handle_rewatch: client blocklisted 2024-09-12T08:50:00.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d25b1c640 -1 librbd::ImageWatcher: 0x5587e9962600 image watch failed: 94042217594752, (107) Transport endpoint is not connected 2024-09-12T08:50:00.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d25b1c640 -1 librbd::Watcher: 0x5587e9962600 handle_error: handle=94042217594752: (107) Transport endpoint is not connected 2024-09-12T08:50:00.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e8b72730 handle_unwatch client blocklisted 2024-09-12T08:50:00.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9962600 handle_rewatch: client blocklisted 2024-09-12T08:50:00.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e956e600 image watch failed: 94042212404480, (107) Transport endpoint is not connected 2024-09-12T08:50:00.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e956e600 handle_error: handle=94042212404480: (107) Transport endpoint is not connected 2024-09-12T08:50:00.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e9cf2600 image watch failed: 94042217600512, (107) Transport endpoint is not connected 2024-09-12T08:50:00.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9cf2600 handle_error: handle=94042217600512: (107) Transport endpoint is not connected 2024-09-12T08:50:00.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e8b72d70 handle_unwatch client blocklisted 2024-09-12T08:50:00.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e956e600 handle_rewatch: client blocklisted 2024-09-12T08:50:00.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e8b72320 handle_unwatch client blocklisted 2024-09-12T08:50:00.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.679+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9cf2600 handle_rewatch: client blocklisted 2024-09-12T08:50:00.689 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.687+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e9a54c00 image watch failed: 94042217595904, (107) Transport endpoint is not connected 2024-09-12T08:50:00.690 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.687+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9a54c00 handle_error: handle=94042217595904: (107) Transport endpoint is not connected 2024-09-12T08:50:00.709 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.707+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e94248c0 handle_unwatch client blocklisted 2024-09-12T08:50:00.709 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.707+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e9a54c00 handle_rewatch: client blocklisted 2024-09-12T08:50:00.710 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.707+0000 7f9d2631d640 -1 librbd::ImageWatcher: 0x5587e956ec00 image watch failed: 94042217598208, (107) Transport endpoint is not connected 2024-09-12T08:50:00.711 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.707+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e956ec00 handle_error: handle=94042217598208: (107) Transport endpoint is not connected 2024-09-12T08:50:00.711 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.707+0000 7f9d2631d640 -1 librbd::watcher::RewatchRequest: 0x5587e8b73ae0 handle_unwatch client blocklisted 2024-09-12T08:50:00.711 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:00.707+0000 7f9d2631d640 -1 librbd::Watcher: 0x5587e956ec00 handle_rewatch: client blocklisted 2024-09-12T08:50:01.658 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/4107428414 until 2024-09-12T09:50:00.855045+0000 (3600 sec) 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d25b1c640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.670 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904f2b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8de4b40 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904e5b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e9ab4780 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904f930 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8c56500 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904ed00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8c56640 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904f1e0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8bb1900 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904e8f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.671 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8bb1a40 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.672 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d25b1c640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904edd0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.672 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8c57400 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.672 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.672 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab5b80 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.672 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.667+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea264000 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.673 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.673 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.673 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.673 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.673 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.673 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab5cc0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea263fc0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab57c0 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8de5e00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab5e00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8da1e00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab5900 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.674 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea264ac0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.675 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea264080 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.675 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea264940 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.675 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea2640c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.675 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.671+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea247c80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.677 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904ea90 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e9ab4140 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904fee0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab4640 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e9729680 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.678 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e8c0baa0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.679 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e916c840 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.679 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e98e4d20 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.679 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e98ae840 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.679 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e97d91a0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.679 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e9ae0960 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587e8bb1a40 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea264880 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::io::AioCompletion: 0x5587ea27ca00 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.680 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d2631d640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587ea263f40 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.681 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e96cbb00 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.681 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e96bed80 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.681 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.681 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5587e904e750 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.681 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.675+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e8da1680 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.682 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.679+0000 7f9d25b1c640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.682 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.679+0000 7f9d25b1c640 -1 librbd::io::AioCompletion: 0x5587e9ab4640 fail: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.683 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.679+0000 7f9d25b1c640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5587e97171c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.684 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.679+0000 7f9d25b1c640 -1 librbd::managed_lock::ReleaseRequest: 0x5587e96fc180 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.686 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.683+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.686 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.683+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.686 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.683+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.686 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.683+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b86e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.690 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.687+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.690 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.687+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea14ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.693 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.691+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.693 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.691+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b86780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.697 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.695+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.697 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.695+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.698 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:50:01.698 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/4107428414 2024-09-12T08:50:01.713 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.707+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.713 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.707+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea175500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.716 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.711+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.716 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.711+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea175a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.737 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.731+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.737 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.731+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b86200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.737 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.731+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.737 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.731+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.746 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.743+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:01.747 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:50:01.743+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:50:02.427 INFO:tasks.workunit.client.0.smithi137.stderr:listed 45 entries 2024-09-12T08:50:02.458 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:50:02.458 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:50:09.041 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:09.035+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:50:09.048 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:09.043+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:50:09.048 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:09.043+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:50:09.048 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:09.043+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:50:12.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:50:12.460 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:50:12.461 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:50:12.462 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:50:13.399 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:50:13.399 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:50:13.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131013 2024-09-12T08:50:13.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:50:13.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4107428414 ]] 2024-09-12T08:50:13.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4107428414 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\0\7\4\2\8\4\1\4 ]] 2024-09-12T08:50:13.400 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:50:21.104 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:50:21.099+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:50:21.104 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:50:21.099+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:50:23.406 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:50:23.409 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:50:23.413 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:50:23.417 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:50:24.477 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:50:24.477 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:50:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131024 2024-09-12T08:50:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:50:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4107428414 ]] 2024-09-12T08:50:24.478 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4107428414 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\0\7\4\2\8\4\1\4 ]] 2024-09-12T08:50:24.479 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:50:28.269 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:50:28.270 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:50:28.262+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:50:34.481 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:50:34.481 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:50:34.481 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:50:34.493 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:50:35.522 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:50:35.523 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:50:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131035 2024-09-12T08:50:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:50:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4107428414 ]] 2024-09-12T08:50:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4107428414 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\0\7\4\2\8\4\1\4 ]] 2024-09-12T08:50:35.524 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:50:45.526 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:50:45.526 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:50:45.528 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:50:45.541 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:50:46.558 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:50:46.559 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:50:46.559 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131046 2024-09-12T08:50:46.559 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:50:46.559 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4107428414 ]] 2024-09-12T08:50:46.559 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4107428414 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\0\7\4\2\8\4\1\4 ]] 2024-09-12T08:50:46.559 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:50:56.562 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:50:56.563 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:50:56.564 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:50:56.576 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:50:57.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/4107428414 2024-09-12T08:50:57.582 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:50:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131057 2024-09-12T08:50:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:50:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/4107428414 ]] 2024-09-12T08:50:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/4107428414 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\0\7\4\2\8\4\1\4 ]] 2024-09-12T08:50:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:51:06.910 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:06.906+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:51:06.920 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:06.918+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:51:06.920 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:06.918+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:51:06.927 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:06.926+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:51:07.593 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:51:07.595 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:51:07.597 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:51:07.618 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:51:08.694 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:51:08.699 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:51:08.700 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131068 2024-09-12T08:51:08.700 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:51:08.700 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:51:08.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\4\1\0\7\4\2\8\4\1\4 ]] 2024-09-12T08:51:08.701 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/313384079 2024-09-12T08:51:10.681 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/313384079 until 2024-09-12T09:51:09.927873+0000 (3600 sec) 2024-09-12T08:51:10.712 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:51:10.712 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/313384079 2024-09-12T08:51:11.429 INFO:tasks.workunit.client.0.smithi137.stderr:listed 46 entries 2024-09-12T08:51:11.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:51:11.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:51:11.547 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.546+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:51:11.550 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:51:11.550 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:51:11.551 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.550+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:51:11.564 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.562+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:51:11.564 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.562+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:51:11.564 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.562+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:51:11.564 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:51:11.562+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:51:21.474 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:51:21.474 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:51:21.478 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:51:21.487 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:51:22.514 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:51:22.514 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:51:22.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131082 2024-09-12T08:51:22.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:51:22.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:51:22.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:51:22.515 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:51:32.520 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:51:32.520 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:51:32.528 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:51:32.539 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:51:33.387 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:51:33.388 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:51:33.388 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131093 2024-09-12T08:51:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:51:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:51:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:51:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:51:43.392 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:51:43.397 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:51:43.398 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:51:43.403 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:51:44.289 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:51:44.290 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:51:44.290 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131104 2024-09-12T08:51:44.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:51:44.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:51:44.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:51:44.291 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:51:54.293 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:51:54.293 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:51:54.294 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:51:54.295 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:51:55.295 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:51:55.295 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:51:55.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131115 2024-09-12T08:51:55.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:51:55.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:51:55.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:51:55.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:52:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.005+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea123f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.005+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.005+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.008 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.005+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9e95f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.009+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.009+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.013+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.013+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.033+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.034 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.033+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea093c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.037+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.037+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.041+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.042 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.041+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.041+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.041+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea329f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.045+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.045+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.049+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.049+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.054 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.053+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.053+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.057+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.057+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.061+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.063 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.061+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.065+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.065+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.069+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.069+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.073+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.073+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.077+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.077+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8916f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.081+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:52:00.081+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8916600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:52:05.297 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:52:05.297 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:52:05.299 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:52:05.301 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:52:06.304 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:52:06.304 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:52:06.304 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131126 2024-09-12T08:52:06.304 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:52:06.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:52:06.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:52:06.305 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:52:08.772 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.765+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:52:08.774 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.773+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:52:08.777 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.773+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:52:08.777 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.773+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:52:08.777 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.773+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:52:08.777 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.773+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:52:08.780 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.777+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:52:08.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.781+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:52:08.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.781+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:52:08.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.781+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:52:08.783 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:08.781+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:52:16.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:52:16.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:52:16.319 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:52:16.333 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:52:17.389 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/313384079 2024-09-12T08:52:17.389 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:52:17.390 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131137 2024-09-12T08:52:17.390 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:52:17.391 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/313384079 ]] 2024-09-12T08:52:17.391 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/313384079 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:52:17.391 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:52:27.157 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:27.153+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:52:27.158 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:27.157+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:52:27.395 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:52:27.395 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:52:27.395 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:52:27.398 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:52:28.469 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:52:28.469 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:52:28.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131148 2024-09-12T08:52:28.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:52:28.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3301685105 ]] 2024-09-12T08:52:28.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3301685105 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\1\3\3\8\4\0\7\9 ]] 2024-09-12T08:52:28.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/3301685105 2024-09-12T08:52:30.704 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/3301685105 until 2024-09-12T09:52:29.924497+0000 (3600 sec) 2024-09-12T08:52:30.734 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:52:30.734 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/3301685105 2024-09-12T08:52:31.558 INFO:tasks.workunit.client.0.smithi137.stderr:listed 47 entries 2024-09-12T08:52:31.598 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:52:31.598 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:52:41.606 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:52:41.607 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:52:41.607 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:52:41.612 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:52:42.672 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:52:42.672 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:52:42.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131162 2024-09-12T08:52:42.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:52:42.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3301685105 ]] 2024-09-12T08:52:42.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3301685105 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\3\0\1\6\8\5\1\0\5 ]] 2024-09-12T08:52:42.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:52:47.914 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:47.908+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:52:47.914 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:47.908+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:52:47.914 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:47.908+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:52:47.922 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:47.920+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:52:47.922 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:47.920+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:52:47.923 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:52:47.920+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:52:48.074 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:52:48.072+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:52:48.074 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:52:48.072+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:52:52.675 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:52:52.677 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:52:52.677 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:52:52.679 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:52:53.657 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:52:53.658 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:52:53.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131173 2024-09-12T08:52:53.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:52:53.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3301685105 ]] 2024-09-12T08:52:53.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3301685105 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\3\0\1\6\8\5\1\0\5 ]] 2024-09-12T08:52:53.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:53:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.000+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.000+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d59080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.004+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.004+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9e95f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.008+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.008+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea160c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.012+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.015 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.012+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9e95980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.020+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.020+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f26b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.024+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.025 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.024+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f21f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.024+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.028 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.024+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea175d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.028+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.028+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.036+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.036+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b86e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.044+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.048 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.044+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b87a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.052+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.052+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b86880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.056+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.056+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea174e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.064+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.068 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.064+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea14ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.072+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.072+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b87d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.080 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.076+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.081 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.076+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9b87500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.084+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.084+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.088+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.091 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.088+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e7280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.096+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.096+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea329d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.104 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.100+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.104 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.100+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.108+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:00.110 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:00.108+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e6f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:53:03.678 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:53:03.688 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:53:03.695 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:53:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:53:04.671 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:53:04.672 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:53:04.672 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131184 2024-09-12T08:53:04.672 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:53:04.672 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3301685105 ]] 2024-09-12T08:53:04.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3301685105 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\3\0\1\6\8\5\1\0\5 ]] 2024-09-12T08:53:04.673 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:53:12.692 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.688+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:53:12.694 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.692+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:53:12.694 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.692+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:53:12.703 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.700+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:53:12.717 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.712+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:53:12.717 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.712+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:53:12.717 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.712+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:53:12.717 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:12.712+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:53:14.674 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:53:14.676 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:53:14.680 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:53:14.683 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:53:15.600 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:53:15.600 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:53:15.600 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131195 2024-09-12T08:53:15.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:53:15.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3301685105 ]] 2024-09-12T08:53:15.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3301685105 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\3\0\1\6\8\5\1\0\5 ]] 2024-09-12T08:53:15.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:53:25.604 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:53:25.609 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:53:25.609 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:53:25.609 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:53:26.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/3301685105 2024-09-12T08:53:26.577 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:53:26.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131206 2024-09-12T08:53:26.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:53:26.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/3301685105 ]] 2024-09-12T08:53:26.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/3301685105 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\3\0\1\6\8\5\1\0\5 ]] 2024-09-12T08:53:26.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:53:30.525 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:30.519+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:53:30.528 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:30.519+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:53:34.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:53:34.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:53:34.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:53:34.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:53:34.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:53:34.966 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.959+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:53:34.973 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.967+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:53:34.973 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.967+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:53:34.974 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.967+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:53:34.974 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.967+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:53:34.974 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:53:34.967+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:53:36.580 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:53:36.581 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:53:36.581 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:53:36.583 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:53:37.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:53:37.566 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:53:37.570 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131217 2024-09-12T08:53:37.570 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:53:37.570 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1366747705 ]] 2024-09-12T08:53:37.570 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1366747705 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\3\3\0\1\6\8\5\1\0\5 ]] 2024-09-12T08:53:37.570 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1366747705 2024-09-12T08:53:39.680 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1366747705 until 2024-09-12T09:53:38.910693+0000 (3600 sec) 2024-09-12T08:53:39.724 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:53:39.724 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1366747705 2024-09-12T08:53:40.522 INFO:tasks.workunit.client.0.smithi137.stderr:listed 48 entries 2024-09-12T08:53:40.571 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:53:40.572 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:53:50.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:53:50.575 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:53:50.575 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:53:50.596 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:53:51.624 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:53:51.624 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:53:51.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131231 2024-09-12T08:53:51.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:53:51.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1366747705 ]] 2024-09-12T08:53:51.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1366747705 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\3\6\6\7\4\7\7\0\5 ]] 2024-09-12T08:53:51.625 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:54:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:59.999+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.004 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:53:59.999+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9617100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.003+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.003+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.007+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.007+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea31da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.011+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.011+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea122c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.019+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.022 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.019+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea31cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.029 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.027+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.027+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.043 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.043+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.046 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.043+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.047+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.047+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.051+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.051+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.057 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.059+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.062 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.059+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea161480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.065 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9e94d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.067+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.067+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.078 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.075+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.079 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.075+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.083 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:54:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:54:01.634 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:54:01.635 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:54:01.635 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:54:01.636 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:54:02.527 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:54:02.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:54:02.527 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131242 2024-09-12T08:54:02.528 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:54:02.528 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1366747705 ]] 2024-09-12T08:54:02.528 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1366747705 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\3\6\6\7\4\7\7\0\5 ]] 2024-09-12T08:54:02.528 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:54:12.530 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:54:12.541 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:54:12.542 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:54:12.550 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:54:13.541 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:54:13.541 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:54:13.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131253 2024-09-12T08:54:13.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:54:13.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1366747705 ]] 2024-09-12T08:54:13.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1366747705 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\3\6\6\7\4\7\7\0\5 ]] 2024-09-12T08:54:13.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:54:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:54:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:54:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:54:23.563 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:54:24.480 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:54:24.480 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:54:24.481 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131264 2024-09-12T08:54:24.481 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:54:24.481 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1366747705 ]] 2024-09-12T08:54:24.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1366747705 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\3\6\6\7\4\7\7\0\5 ]] 2024-09-12T08:54:24.482 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:54:34.483 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:54:34.483 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:54:34.484 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:54:34.485 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:54:35.538 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1366747705 2024-09-12T08:54:35.541 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:54:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131275 2024-09-12T08:54:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:54:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1366747705 ]] 2024-09-12T08:54:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1366747705 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\3\6\6\7\4\7\7\0\5 ]] 2024-09-12T08:54:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:54:45.549 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:54:45.550 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:54:45.556 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:54:45.567 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:54:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:54:46.609 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:54:46.609 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131286 2024-09-12T08:54:46.609 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:54:46.609 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2808455350 ]] 2024-09-12T08:54:46.609 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2808455350 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\3\6\6\7\4\7\7\0\5 ]] 2024-09-12T08:54:46.610 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2808455350 2024-09-12T08:54:48.680 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2808455350 until 2024-09-12T09:54:48.001662+0000 (3600 sec) 2024-09-12T08:54:48.715 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:54:48.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2808455350 2024-09-12T08:54:49.473 INFO:tasks.workunit.client.0.smithi137.stderr:listed 49 entries 2024-09-12T08:54:49.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:54:49.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:54:54.589 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:54:54.582+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:54:59.526 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:54:59.526 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:54:59.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:54:59.527 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:55:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.002+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.005 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.002+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9f27b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.006+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.006+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587ea123700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.010+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.010+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9e95380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.018+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.018+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e90ef780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.030+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.030 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.030+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9e94800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.038+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.038+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.042+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.042+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e99e7500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.052 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.050+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.050+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.062+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.062+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31ca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.066+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.069 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.066+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.074+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.074 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.074+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9cfaf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.085 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.082+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.086 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.082+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a5f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.094+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.096 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.094+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.098+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.099 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.098+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.106+0000 7f9d2631d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.109 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.110+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.113 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.110+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.113 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.110+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.114+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.114+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.123 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.122+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.123 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.122+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.129 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.126+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.129 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.126+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.131 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.130+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.131 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:55:00.130+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea328d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:55:00.492 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:55:00.492 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:55:00.493 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131300 2024-09-12T08:55:00.493 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:55:00.493 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2808455350 ]] 2024-09-12T08:55:00.493 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2808455350 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\8\0\8\4\5\5\3\5\0 ]] 2024-09-12T08:55:00.493 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:55:10.505 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:55:10.505 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:55:10.505 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:55:10.509 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:55:11.533 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:55:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:55:11.548 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131311 2024-09-12T08:55:11.548 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:55:11.548 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2808455350 ]] 2024-09-12T08:55:11.548 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2808455350 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\8\0\8\4\5\5\3\5\0 ]] 2024-09-12T08:55:11.549 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:55:21.553 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:55:21.553 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:55:21.554 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:55:21.554 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:55:22.678 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:55:22.683 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:55:22.684 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131322 2024-09-12T08:55:22.684 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:55:22.684 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2808455350 ]] 2024-09-12T08:55:22.684 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2808455350 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\8\0\8\4\5\5\3\5\0 ]] 2024-09-12T08:55:22.684 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:55:32.685 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:55:32.688 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:55:32.688 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:55:32.690 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:55:33.629 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:55:33.629 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:55:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131333 2024-09-12T08:55:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:55:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2808455350 ]] 2024-09-12T08:55:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2808455350 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\8\0\8\4\5\5\3\5\0 ]] 2024-09-12T08:55:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:55:35.878 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:55:35.873+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:55:35.878 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:55:35.873+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:55:35.898 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:55:35.893+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:55:43.633 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:55:43.633 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:55:43.633 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:55:43.647 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:55:44.651 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2808455350 2024-09-12T08:55:44.653 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:55:44.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131344 2024-09-12T08:55:44.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:55:44.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2808455350 ]] 2024-09-12T08:55:44.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2808455350 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\8\0\8\4\5\5\3\5\0 ]] 2024-09-12T08:55:44.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:55:54.669 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:55:54.669 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:55:54.673 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:55:54.687 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:55:55.657 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:55:55.657 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:55:55.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131355 2024-09-12T08:55:55.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:55:55.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1860195745 ]] 2024-09-12T08:55:55.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1860195745 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\8\0\8\4\5\5\3\5\0 ]] 2024-09-12T08:55:55.658 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/1860195745 2024-09-12T08:55:57.681 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/1860195745 until 2024-09-12T09:55:56.940663+0000 (3600 sec) 2024-09-12T08:55:57.715 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:55:57.715 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/1860195745 2024-09-12T08:55:58.533 INFO:tasks.workunit.client.0.smithi137.stderr:listed 50 entries 2024-09-12T08:55:58.572 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:55:58.572 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:56:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.010 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e7200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.009+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.013+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.016 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.013+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.017+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.021 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.017+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9972300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.023 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.021+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.021+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.025+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.025+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.029+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.031 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.029+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea093f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.033+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.035 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.033+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea160c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.037+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.037+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.041+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.041 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.041+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.041+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.041+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea175200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.045+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.047 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.045+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.049+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.049 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.049+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.049+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.053 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.049+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea31d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.053+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.053+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.060 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.057+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.061 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.057+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea175d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.061+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.061+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0a4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.065+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:00.067 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:56:00.065+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:56:08.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:56:08.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:56:08.574 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:56:08.579 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:56:09.504 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:56:09.504 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:56:09.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131369 2024-09-12T08:56:09.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:56:09.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1860195745 ]] 2024-09-12T08:56:09.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1860195745 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\8\6\0\1\9\5\7\4\5 ]] 2024-09-12T08:56:09.505 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:56:19.514 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:56:19.516 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:56:19.518 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:56:19.521 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:56:20.561 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:56:20.561 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:56:20.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131380 2024-09-12T08:56:20.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:56:20.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1860195745 ]] 2024-09-12T08:56:20.562 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1860195745 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\8\6\0\1\9\5\7\4\5 ]] 2024-09-12T08:56:20.563 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:56:30.569 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:56:30.575 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:56:30.586 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:56:30.590 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:56:31.618 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:56:31.618 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:56:31.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131391 2024-09-12T08:56:31.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:56:31.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1860195745 ]] 2024-09-12T08:56:31.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1860195745 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\8\6\0\1\9\5\7\4\5 ]] 2024-09-12T08:56:31.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:56:39.093 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:56:39.088+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:56:41.628 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:56:41.628 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:56:41.628 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:56:41.634 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:56:42.736 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:56:42.736 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:56:42.737 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131402 2024-09-12T08:56:42.737 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:56:42.737 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1860195745 ]] 2024-09-12T08:56:42.737 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1860195745 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\8\6\0\1\9\5\7\4\5 ]] 2024-09-12T08:56:42.737 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:56:42.970 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:56:42.968+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:56:42.970 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:56:42.968+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:56:52.741 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:56:52.741 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:56:52.741 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:56:52.758 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:56:53.601 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/1860195745 2024-09-12T08:56:53.602 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:56:53.602 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131413 2024-09-12T08:56:53.602 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:56:53.602 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/1860195745 ]] 2024-09-12T08:56:53.602 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/1860195745 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\8\6\0\1\9\5\7\4\5 ]] 2024-09-12T08:56:53.602 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:57:03.613 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:57:03.614 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:57:03.615 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:57:03.622 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:57:04.612 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:57:04.612 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:57:04.613 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131424 2024-09-12T08:57:04.613 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:57:04.614 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/640503035 ]] 2024-09-12T08:57:04.614 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/640503035 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\1\8\6\0\1\9\5\7\4\5 ]] 2024-09-12T08:57:04.614 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/640503035 2024-09-12T08:57:06.786 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/640503035 until 2024-09-12T09:57:05.981271+0000 (3600 sec) 2024-09-12T08:57:06.827 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/640503035 2024-09-12T08:57:06.837 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:57:07.589 INFO:tasks.workunit.client.0.smithi137.stderr:listed 51 entries 2024-09-12T08:57:07.632 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:57:07.632 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:57:17.633 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:57:17.634 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:57:17.635 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:57:17.637 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:57:18.576 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:57:18.576 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:57:18.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131438 2024-09-12T08:57:18.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:57:18.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/640503035 ]] 2024-09-12T08:57:18.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/640503035 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\4\0\5\0\3\0\3\5 ]] 2024-09-12T08:57:18.577 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:57:22.265 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.259+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:57:22.277 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.271+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:57:22.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.275+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:57:22.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.275+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:57:22.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.275+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:57:22.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.279+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:57:22.286 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.279+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:57:22.287 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.283+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:57:22.287 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.283+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:57:22.287 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:22.283+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:57:28.579 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:57:28.579 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:57:28.580 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:57:28.581 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:57:29.518 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:57:29.519 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:57:29.519 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131449 2024-09-12T08:57:29.520 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:57:29.520 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/640503035 ]] 2024-09-12T08:57:29.520 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/640503035 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\4\0\5\0\3\0\3\5 ]] 2024-09-12T08:57:29.520 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:57:33.944 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:57:33.944 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:57:33.944 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:57:33.944 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:57:33.944 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:57:33.945 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:57:33.945 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:57:33.945 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:57:33.945 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:57:33.945 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:57:33.946 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:57:33.946 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:57:33.946 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:33.935+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:57:39.525 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:57:39.525 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:57:39.550 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:57:39.559 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:57:40.647 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:57:40.647 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:57:40.647 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131460 2024-09-12T08:57:40.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:57:40.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/640503035 ]] 2024-09-12T08:57:40.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/640503035 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\4\0\5\0\3\0\3\5 ]] 2024-09-12T08:57:40.648 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:57:49.173 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.167+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:57:49.178 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.171+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:57:49.178 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:57:49.178 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:57:49.179 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:57:49.179 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.175+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:57:49.183 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.179+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:57:49.194 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.191+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:57:49.194 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.191+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:57:49.194 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.191+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:57:49.194 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:57:49.191+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:57:50.655 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:57:50.656 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:57:50.662 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:57:50.664 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:57:51.649 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:57:51.650 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:57:51.656 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131471 2024-09-12T08:57:51.656 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:57:51.656 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/640503035 ]] 2024-09-12T08:57:51.656 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/640503035 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\4\0\5\0\3\0\3\5 ]] 2024-09-12T08:57:51.656 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:58:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.006 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e6180 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.003+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.003+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.007+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.012 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.007+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86500 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.015+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.015+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b86200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.019+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.027 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.023+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9b87b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.027+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.027+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174000 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.035+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.038 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.035+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.039+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.039+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea175d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.055 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.051+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.056 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.051+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea174e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.055+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.055+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea14ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.063+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.066 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.063+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e95a8e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.071+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.071+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.079+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.079+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916400 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.083+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.087 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.083+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8916980 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.090 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e8917300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.087+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.087+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.091+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.095 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.091+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0ea780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.095+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.095+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0eb800 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.099+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.099+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9d96880 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.103+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:58:00.103+0000 7f9d2631d640 -1 librbd::ImageState: 0x5587e9d96a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:58:01.666 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:58:01.666 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:58:01.671 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:58:01.673 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:58:02.738 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/640503035 2024-09-12T08:58:02.739 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:58:02.739 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131482 2024-09-12T08:58:02.740 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:58:02.740 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/640503035 ]] 2024-09-12T08:58:02.740 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/640503035 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\4\0\5\0\3\0\3\5 ]] 2024-09-12T08:58:02.740 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:58:12.746 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:58:12.746 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:58:12.751 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:58:12.759 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:58:13.751 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2502491016 2024-09-12T08:58:13.751 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:58:13.752 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131493 2024-09-12T08:58:13.752 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:58:13.753 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2502491016 ]] 2024-09-12T08:58:13.753 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2502491016 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\6\4\0\5\0\3\0\3\5 ]] 2024-09-12T08:58:13.753 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist add 172.21.15.137:0/2502491016 2024-09-12T08:58:13.833 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:13.827+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:58:13.833 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:13.827+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:58:15.787 INFO:tasks.workunit.client.0.smithi137.stderr:blocklisting 172.21.15.137:0/2502491016 until 2024-09-12T09:58:15.031660+0000 (3600 sec) 2024-09-12T08:58:15.852 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep -q 172.21.15.137:0/2502491016 2024-09-12T08:58:15.852 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph osd blocklist ls 2024-09-12T08:58:16.683 INFO:tasks.workunit.client.0.smithi137.stderr:listed 52 entries 2024-09-12T08:58:16.725 INFO:tasks.workunit.client.0.smithi137.stderr:+ PREV_CLIENT_ADDR=172.21.15.137:0/2502491016 2024-09-12T08:58:16.725 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:58:20.720 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:58:20.714+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a2eca5b0 2024-09-12T08:58:20.721 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-12T08:58:20.714+0000 7f2a596e2640 -1 reset not still connected to 0x55f7a3c145b0 2024-09-12T08:58:20.736 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.730+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:58:20.740 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.730+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:58:20.751 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:58:20.751 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:58:20.751 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:58:20.751 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:58:20.751 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:58:20.752 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:58:20.752 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:58:20.752 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:58:20.752 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:58:20.752 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:20.746+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:58:26.726 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:58:26.729 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:58:26.729 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:58:26.738 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:58:27.719 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2502491016 2024-09-12T08:58:27.719 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:58:27.720 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131507 2024-09-12T08:58:27.720 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:58:27.720 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2502491016 ]] 2024-09-12T08:58:27.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2502491016 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\0\2\4\9\1\0\1\6 ]] 2024-09-12T08:58:27.721 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:58:37.577 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:37.566+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:58:37.577 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:37.570+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:58:37.586 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:37.582+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:58:37.596 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:37.586+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:58:37.601 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:58:37.598+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:58:37.726 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:58:37.726 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:58:37.733 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:58:37.743 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:58:38.657 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2502491016 2024-09-12T08:58:38.665 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:58:38.667 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131518 2024-09-12T08:58:38.667 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:58:38.667 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2502491016 ]] 2024-09-12T08:58:38.667 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2502491016 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\0\2\4\9\1\0\1\6 ]] 2024-09-12T08:58:38.667 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:58:48.669 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:58:48.669 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:58:48.669 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:58:48.684 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:58:49.605 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2502491016 2024-09-12T08:58:49.605 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:58:49.606 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131529 2024-09-12T08:58:49.606 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:58:49.606 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ -n 172.21.15.137:0/2502491016 ]] 2024-09-12T08:58:49.606 INFO:tasks.workunit.client.0.smithi137.stderr:+ [[ 172.21.15.137:0/2502491016 != \1\7\2\.\2\1\.\1\5\.\1\3\7\:\0\/\2\5\0\2\4\9\1\0\1\6 ]] 2024-09-12T08:58:49.606 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:58:59.610 INFO:tasks.workunit.client.0.smithi137.stderr:++ ceph mgr dump 2024-09-12T08:58:59.612 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-12T08:58:59.612 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-12T08:58:59.625 INFO:tasks.workunit.client.0.smithi137.stderr:++ jq '.active_clients[]' 2024-09-12T08:59:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.006+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.009 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.006+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea0a4c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.010+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.011 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.010+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9616080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.014+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.014 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.014+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.019 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.018+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.020 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.018+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e9617080 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.030+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.032 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.030+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea31d380 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.046+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.045 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.046+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587ea328480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.058+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.059 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.058+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e99e7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.070+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.070 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.070+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea329900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.074+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.077 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.074+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.082+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.082 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.082+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.086+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.088 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.086+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.090+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.089 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.090+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9f20b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.090+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.090+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9b86280 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.098+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.098+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9e95480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.102+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.102 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.102+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e99e6b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.106+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.107 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.106+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8917580 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.111 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.110+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.111 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.110+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e8917200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.114+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.116 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.114+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587ea0eb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.118+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.118+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96300 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.125 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.122+0000 7f9d2531b640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.125 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.122+0000 7f9d2531b640 -1 librbd::ImageState: 0x5587e9d96200 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.499 INFO:tasks.workunit.client.0.smithi137.stderr:+ CLIENT_ADDR=172.21.15.137:0/2502491016 2024-09-12T08:59:00.500 INFO:tasks.workunit.client.0.smithi137.stderr:++ date +%s 2024-09-12T08:59:00.502 INFO:tasks.workunit.client.0.smithi137.stderr:+ CURRENT_TIME=1726131540 2024-09-12T08:59:00.502 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( CURRENT_TIME <= END_TIME )) 2024-09-12T08:59:00.502 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i = 1 )) 2024-09-12T08:59:00.502 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 24 )) 2024-09-12T08:59:00.502 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2024-09-12T08:59:00.769 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.766+0000 7f9d25b1c640 -1 librbd::image::OpenRequest: failed to stat v2 image header: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.769 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.766+0000 7f9d25b1c640 -1 librbd::ImageState: 0x5587e944e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-12T08:59:00.802 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:00.802+0000 7f9d21b14640 -1 mgr.server reply reply (11) Resource temporarily unavailable [errno 108] RBD connection was shutdown (error opening image b'image1' at snapshot None) 2024-09-12T08:59:00.802 INFO:tasks.workunit.client.0.smithi137.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: [errno 108] RBD connection was shutdown (error opening image b'image1' at snapshot None) 2024-09-12T08:59:00.810 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:59:09.414 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8386dd0 2024-09-12T08:59:09.419 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a83876c0 2024-09-12T08:59:09.419 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:59:09.419 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8bafba0 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a90416c0 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:59:09.420 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:09.414+0000 7fb165145640 -1 reset not still connected to 0x5558a9888c30 2024-09-12T08:59:10.811 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:10.812 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 24 )) 2024-09-12T08:59:10.812 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2024-09-12T08:59:10.918 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:10.918+0000 7f9d21b14640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-12T08:59:10.918 INFO:tasks.workunit.client.0.smithi137.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-12T08:59:10.930 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:59:18.094 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:18.093+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:59:18.094 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:18.093+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:59:19.808 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a8387790 2024-09-12T08:59:19.808 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a8bae750 2024-09-12T08:59:19.808 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a8baed00 2024-09-12T08:59:19.808 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52410 2024-09-12T08:59:19.808 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a8f524e0 2024-09-12T08:59:19.809 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a8f52d00 2024-09-12T08:59:19.809 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a90405b0 2024-09-12T08:59:19.809 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-12T08:59:19.805+0000 7fb165145640 -1 reset not still connected to 0x5558a9321860 2024-09-12T08:59:20.934 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:20.934 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 24 )) 2024-09-12T08:59:20.934 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2024-09-12T08:59:21.024 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T08:59:21.021+0000 7f9d21b14640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-12T08:59:21.025 INFO:tasks.workunit.client.0.smithi137.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-12T08:59:21.031 INFO:tasks.workunit.client.0.smithi137.stderr:+ sleep 10 2024-09-12T08:59:31.033 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:31.033 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 24 )) 2024-09-12T08:59:31.033 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2024-09-12T08:59:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:+ break 2024-09-12T08:59:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2024-09-12T08:59:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep 'every 2m' 2024-09-12T08:59:31.362 INFO:tasks.workunit.client.0.smithi137.stdout:every 1m, every 2m 2024-09-12T08:59:31.363 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2024-09-12T08:59:31.363 INFO:tasks.workunit.client.0.smithi137.stderr:+ grep 'every 1m' 2024-09-12T08:59:31.470 INFO:tasks.workunit.client.0.smithi137.stdout:every 1m, every 2m 2024-09-12T08:59:31.471 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image1 2m 2024-09-12T08:59:31.608 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i = 1 )) 2024-09-12T08:59:31.608 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:31.608 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image1 1m 2024-09-12T08:59:31.777 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:31.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:31.845 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image2 1m 2024-09-12T08:59:31.935 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:31.935 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:31.935 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image3 1m 2024-09-12T08:59:32.057 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.057 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.057 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image4 1m 2024-09-12T08:59:32.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.250 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image5 1m 2024-09-12T08:59:32.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.429 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image6 1m 2024-09-12T08:59:32.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.578 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image7 1m 2024-09-12T08:59:32.694 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.695 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image8 1m 2024-09-12T08:59:32.847 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.848 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.848 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image9 1m 2024-09-12T08:59:32.976 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:32.976 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:32.976 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image10 1m 2024-09-12T08:59:33.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:33.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:33.104 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image11 1m 2024-09-12T08:59:33.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:33.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:33.357 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image12 1m 2024-09-12T08:59:33.511 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:33.511 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:33.511 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image13 1m 2024-09-12T08:59:33.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:33.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:33.654 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image14 1m 2024-09-12T08:59:33.771 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:33.771 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:33.771 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image15 1m 2024-09-12T08:59:33.979 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:33.979 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:33.979 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image16 1m 2024-09-12T08:59:34.101 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image17 1m 2024-09-12T08:59:34.226 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:34.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:34.227 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image18 1m 2024-09-12T08:59:34.422 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:34.422 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:34.422 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image19 1m 2024-09-12T08:59:34.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:34.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:34.622 INFO:tasks.workunit.client.0.smithi137.stderr:+ rbd mirror snapshot schedule rm -p rbd --image image20 1m 2024-09-12T08:59:34.999 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:34.999 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:34.999 INFO:tasks.workunit.client.0.smithi137.stderr:+ killall fio 2024-09-12T08:59:35.049 INFO:tasks.workunit.client.0.smithi137.stderr:+ wait 2024-09-12T08:59:35.181 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i = 1 )) 2024-09-12T08:59:35.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.182 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd0 2024-09-12T08:59:35.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.396 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd1 2024-09-12T08:59:35.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.470 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd2 2024-09-12T08:59:35.525 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.525 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.525 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd3 2024-09-12T08:59:35.595 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.595 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.595 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd4 2024-09-12T08:59:35.664 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.664 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.664 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd5 2024-09-12T08:59:35.734 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.734 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.734 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd6 2024-09-12T08:59:35.802 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.802 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.802 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd7 2024-09-12T08:59:35.870 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.870 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.870 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd8 2024-09-12T08:59:35.942 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:35.942 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:35.942 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd9 2024-09-12T08:59:36.016 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.017 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.017 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd10 2024-09-12T08:59:36.072 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.072 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.072 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd11 2024-09-12T08:59:36.151 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.151 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.152 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd12 2024-09-12T08:59:36.218 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.218 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.218 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd13 2024-09-12T08:59:36.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.296 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd14 2024-09-12T08:59:36.367 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.367 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.368 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd15 2024-09-12T08:59:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.440 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd16 2024-09-12T08:59:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd17 2024-09-12T08:59:36.586 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.586 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.586 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd18 2024-09-12T08:59:36.661 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.661 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.661 INFO:tasks.workunit.client.0.smithi137.stderr:+ sudo rbd device unmap /dev/rbd19 2024-09-12T08:59:36.718 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i++ )) 2024-09-12T08:59:36.719 INFO:tasks.workunit.client.0.smithi137.stderr:+ (( i <= 20 )) 2024-09-12T08:59:36.719 INFO:tasks.workunit.client.0.smithi137.stderr:+ echo OK 2024-09-12T08:59:36.719 INFO:tasks.workunit.client.0.smithi137.stdout:OK 2024-09-12T08:59:36.731 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-12T08:59:36.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-12T08:59:36.821 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2024-09-12T08:59:36.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-12T08:59:37.096 DEBUG:teuthology.parallel:result is None 2024-09-12T08:59:37.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-12T08:59:37.120 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-12T08:59:37.120 DEBUG:teuthology.orchestra.run.smithi137:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-12T08:59:37.184 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-12T08:59:37.184 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-12T08:59:37.231 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-12T08:59:37.240 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-12T08:59:37.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T08:59:37.539 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T08:59:37.540 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T08:59:37.557 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2029,"stamp":"2024-09-12T08:59:36.674964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27648062185,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1768800,"num_read_kb":6742551,"num_write":1697609,"num_write_kb":6787244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85041,"ondisk_log_size":85041,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18892524,"kb_used_data":13573308,"kb_used_omap":2900,"kb_used_meta":5316267,"kb_avail":264222996,"statfs":{"total":289910292480,"available":270564347904,"internally_reserved":0,"allocated":13899067392,"data_stored":21945474271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2970163,"internal_metadata":5443857869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,4,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17747968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5577,"num_read_kb":19016,"num_write":4349,"num_write_kb":17334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004449"},"pg_stats":[{"pgid":"2.7","version":"165'5991","reported_seq":9803,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.075209+0000","last_change":"2024-09-12T07:57:31.593411+0000","last_active":"2024-09-12T08:59:34.075209+0000","last_peered":"2024-09-12T08:59:34.075209+0000","last_clean":"2024-09-12T08:59:34.075209+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:34.075209+0000","last_undegraded":"2024-09-12T08:59:34.075209+0000","last_fullsized":"2024-09-12T08:59:34.075209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:47.434507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6313,"num_read_kb":5114,"num_write":123,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"165'5892","reported_seq":12014,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.406135+0000","last_change":"2024-09-12T07:57:31.592720+0000","last_active":"2024-09-12T08:59:34.406135+0000","last_peered":"2024-09-12T08:59:34.406135+0000","last_clean":"2024-09-12T08:59:34.406135+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:34.406135+0000","last_undegraded":"2024-09-12T08:59:34.406135+0000","last_fullsized":"2024-09-12T08:59:34.406135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:40:49.569427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10704,"num_read_kb":9552,"num_write":126,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"165'8922","reported_seq":14422,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.554788+0000","last_change":"2024-09-12T07:57:31.592903+0000","last_active":"2024-09-12T08:59:34.554788+0000","last_peered":"2024-09-12T08:59:34.554788+0000","last_clean":"2024-09-12T08:59:34.554788+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:34.554788+0000","last_undegraded":"2024-09-12T08:59:34.554788+0000","last_fullsized":"2024-09-12T08:59:34.554788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:31.416374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8864,"num_read_kb":7233,"num_write":172,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'8903","reported_seq":15595,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.267937+0000","last_change":"2024-09-12T07:57:31.592249+0000","last_active":"2024-09-12T08:59:33.267937+0000","last_peered":"2024-09-12T08:59:33.267937+0000","last_clean":"2024-09-12T08:59:33.267937+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:33.267937+0000","last_undegraded":"2024-09-12T08:59:33.267937+0000","last_fullsized":"2024-09-12T08:59:33.267937+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8903,"log_dups_size":0,"ondisk_log_size":8903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:43:47.133587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10611,"num_read_kb":8785,"num_write":168,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"165'432146","reported_seq":858260,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.837025+0000","last_change":"2024-09-12T08:50:01.805419+0000","last_active":"2024-09-12T08:59:33.837025+0000","last_peered":"2024-09-12T08:59:33.837025+0000","last_clean":"2024-09-12T08:59:33.837025+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T08:59:33.837025+0000","last_undegraded":"2024-09-12T08:59:33.837025+0000","last_fullsized":"2024-09-12T08:59:33.837025+0000","mapping_epoch":17,"log_start":"162'422142","ondisk_log_start":"162'422142","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:40:29.170563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6986854400,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429419,"num_read_kb":1688160,"num_write":428383,"num_write_kb":1713532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"165'5935","reported_seq":11177,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.748593+0000","last_change":"2024-09-12T07:57:31.586790+0000","last_active":"2024-09-12T08:59:33.748593+0000","last_peered":"2024-09-12T08:59:33.748593+0000","last_clean":"2024-09-12T08:59:33.748593+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T08:59:33.748593+0000","last_undegraded":"2024-09-12T08:59:33.748593+0000","last_fullsized":"2024-09-12T08:59:33.748593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:26:12.712604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7660,"num_read_kb":6527,"num_write":159,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"165'429312","reported_seq":852731,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.592310+0000","last_change":"2024-09-12T08:50:01.794946+0000","last_active":"2024-09-12T08:59:34.592310+0000","last_peered":"2024-09-12T08:59:34.592310+0000","last_clean":"2024-09-12T08:59:34.592310+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T08:59:34.592310+0000","last_undegraded":"2024-09-12T08:59:34.592310+0000","last_fullsized":"2024-09-12T08:59:34.592310+0000","mapping_epoch":17,"log_start":"162'419236","ondisk_log_start":"162'419236","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:42:22.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930223104,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426680,"num_read_kb":1677524,"num_write":425593,"num_write_kb":1702372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'145","reported_seq":3677,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:32.200034+0000","last_change":"2024-09-12T07:57:31.577407+0000","last_active":"2024-09-12T08:59:32.200034+0000","last_peered":"2024-09-12T08:59:32.200034+0000","last_clean":"2024-09-12T08:59:32.200034+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:32.200034+0000","last_undegraded":"2024-09-12T08:59:32.200034+0000","last_fullsized":"2024-09-12T08:59:32.200034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:26:14.289003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5919,"num_read_kb":4807,"num_write":86,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"165'428859","reported_seq":851077,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.835734+0000","last_change":"2024-09-12T08:50:01.747537+0000","last_active":"2024-09-12T08:59:33.835734+0000","last_peered":"2024-09-12T08:59:33.835734+0000","last_clean":"2024-09-12T08:59:33.835734+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T08:59:33.835734+0000","last_undegraded":"2024-09-12T08:59:33.835734+0000","last_fullsized":"2024-09-12T08:59:33.835734+0000","mapping_epoch":17,"log_start":"162'418842","ondisk_log_start":"162'418842","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:34:43.580846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6927671296,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425473,"num_read_kb":1671764,"num_write":425145,"num_write_kb":1700580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'242","reported_seq":5444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:32.403230+0000","last_change":"2024-09-12T07:57:31.577181+0000","last_active":"2024-09-12T08:59:32.403230+0000","last_peered":"2024-09-12T08:59:32.403230+0000","last_clean":"2024-09-12T08:59:32.403230+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:32.403230+0000","last_undegraded":"2024-09-12T08:59:32.403230+0000","last_fullsized":"2024-09-12T08:59:32.403230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:59:10.221338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8509,"num_read_kb":6945,"num_write":134,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"165'420958","reported_seq":837362,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.600802+0000","last_change":"2024-09-12T08:50:01.692389+0000","last_active":"2024-09-12T08:59:34.600802+0000","last_peered":"2024-09-12T08:59:34.600802+0000","last_clean":"2024-09-12T08:59:34.600802+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T08:59:34.600802+0000","last_undegraded":"2024-09-12T08:59:34.600802+0000","last_fullsized":"2024-09-12T08:59:34.600802+0000","mapping_epoch":17,"log_start":"162'410926","ondisk_log_start":"162'410926","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:58:52.087120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802718739,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419602,"num_read_kb":1648780,"num_write":417305,"num_write_kb":1669214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"165'8830","reported_seq":14099,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.173699+0000","last_change":"2024-09-12T07:57:31.592415+0000","last_active":"2024-09-12T08:59:34.173699+0000","last_peered":"2024-09-12T08:59:34.173699+0000","last_clean":"2024-09-12T08:59:34.173699+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:34.173699+0000","last_undegraded":"2024-09-12T08:59:34.173699+0000","last_fullsized":"2024-09-12T08:59:34.173699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:25:45.764022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8990,"num_read_kb":7314,"num_write":136,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":416,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:13.335552+0000","last_change":"2024-09-12T07:57:26.399925+0000","last_active":"2024-09-12T08:59:13.335552+0000","last_peered":"2024-09-12T08:59:13.335552+0000","last_clean":"2024-09-12T08:59:13.335552+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:13.335552+0000","last_undegraded":"2024-09-12T08:59:13.335552+0000","last_fullsized":"2024-09-12T08:59:13.335552+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_clean_scrub_stamp":"2024-09-12T07:57:24.546601+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:46:35.118087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27647467539,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701174,"num_read_kb":6686228,"num_write":1696426,"num_write_kb":6785698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13894971392,"data_stored":21945020454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40129,"ondisk_log_size":40129,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67570,"num_read_kb":56277,"num_write":1104,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46711,"internal_metadata":0},"log_size":44860,"ondisk_log_size":44860,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706415,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8889128,"kb_used_data":6783832,"kb_used_omap":1495,"kb_used_meta":2103784,"kb_avail":85482712,"statfs":{"total":96636764160,"available":87534297088,"internally_reserved":0,"allocated":6946643968,"data_stored":10970582489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1531196,"internal_metadata":2154275524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706413,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4968264,"kb_used_data":3373432,"kb_used_omap":700,"kb_used_meta":1594115,"kb_avail":89403576,"statfs":{"total":96636764160,"available":91549261824,"internally_reserved":0,"allocated":3454394368,"data_stored":5453281944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1632374214},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5035132,"kb_used_data":3416044,"kb_used_omap":704,"kb_used_meta":1618367,"kb_avail":89336708,"statfs":{"total":96636764160,"available":91480788992,"internally_reserved":0,"allocated":3498029056,"data_stored":5521609838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1657208131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,7,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12848,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3496067072,"data_stored":5520855040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452526592,"data_stored":5452763155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6946377728,"data_stored":10971402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T08:59:37.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T08:59:37.847 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T08:59:37.847 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T08:59:37.864 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2029,"stamp":"2024-09-12T08:59:36.674964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27648062185,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1768800,"num_read_kb":6742551,"num_write":1697609,"num_write_kb":6787244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85041,"ondisk_log_size":85041,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18892524,"kb_used_data":13573308,"kb_used_omap":2900,"kb_used_meta":5316267,"kb_avail":264222996,"statfs":{"total":289910292480,"available":270564347904,"internally_reserved":0,"allocated":13899067392,"data_stored":21945474271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2970163,"internal_metadata":5443857869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,4,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17747968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5577,"num_read_kb":19016,"num_write":4349,"num_write_kb":17334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004449"},"pg_stats":[{"pgid":"2.7","version":"165'5991","reported_seq":9803,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.075209+0000","last_change":"2024-09-12T07:57:31.593411+0000","last_active":"2024-09-12T08:59:34.075209+0000","last_peered":"2024-09-12T08:59:34.075209+0000","last_clean":"2024-09-12T08:59:34.075209+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:34.075209+0000","last_undegraded":"2024-09-12T08:59:34.075209+0000","last_fullsized":"2024-09-12T08:59:34.075209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:47.434507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6313,"num_read_kb":5114,"num_write":123,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"165'5892","reported_seq":12014,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.406135+0000","last_change":"2024-09-12T07:57:31.592720+0000","last_active":"2024-09-12T08:59:34.406135+0000","last_peered":"2024-09-12T08:59:34.406135+0000","last_clean":"2024-09-12T08:59:34.406135+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:34.406135+0000","last_undegraded":"2024-09-12T08:59:34.406135+0000","last_fullsized":"2024-09-12T08:59:34.406135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:40:49.569427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10704,"num_read_kb":9552,"num_write":126,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"165'8922","reported_seq":14422,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.554788+0000","last_change":"2024-09-12T07:57:31.592903+0000","last_active":"2024-09-12T08:59:34.554788+0000","last_peered":"2024-09-12T08:59:34.554788+0000","last_clean":"2024-09-12T08:59:34.554788+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:34.554788+0000","last_undegraded":"2024-09-12T08:59:34.554788+0000","last_fullsized":"2024-09-12T08:59:34.554788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:31.416374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8864,"num_read_kb":7233,"num_write":172,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'8903","reported_seq":15595,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.267937+0000","last_change":"2024-09-12T07:57:31.592249+0000","last_active":"2024-09-12T08:59:33.267937+0000","last_peered":"2024-09-12T08:59:33.267937+0000","last_clean":"2024-09-12T08:59:33.267937+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:33.267937+0000","last_undegraded":"2024-09-12T08:59:33.267937+0000","last_fullsized":"2024-09-12T08:59:33.267937+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8903,"log_dups_size":0,"ondisk_log_size":8903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:43:47.133587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10611,"num_read_kb":8785,"num_write":168,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"165'432146","reported_seq":858260,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.837025+0000","last_change":"2024-09-12T08:50:01.805419+0000","last_active":"2024-09-12T08:59:33.837025+0000","last_peered":"2024-09-12T08:59:33.837025+0000","last_clean":"2024-09-12T08:59:33.837025+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T08:59:33.837025+0000","last_undegraded":"2024-09-12T08:59:33.837025+0000","last_fullsized":"2024-09-12T08:59:33.837025+0000","mapping_epoch":17,"log_start":"162'422142","ondisk_log_start":"162'422142","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:40:29.170563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6986854400,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429419,"num_read_kb":1688160,"num_write":428383,"num_write_kb":1713532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"165'5935","reported_seq":11177,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.748593+0000","last_change":"2024-09-12T07:57:31.586790+0000","last_active":"2024-09-12T08:59:33.748593+0000","last_peered":"2024-09-12T08:59:33.748593+0000","last_clean":"2024-09-12T08:59:33.748593+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T08:59:33.748593+0000","last_undegraded":"2024-09-12T08:59:33.748593+0000","last_fullsized":"2024-09-12T08:59:33.748593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:26:12.712604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7660,"num_read_kb":6527,"num_write":159,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"165'429312","reported_seq":852731,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.592310+0000","last_change":"2024-09-12T08:50:01.794946+0000","last_active":"2024-09-12T08:59:34.592310+0000","last_peered":"2024-09-12T08:59:34.592310+0000","last_clean":"2024-09-12T08:59:34.592310+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T08:59:34.592310+0000","last_undegraded":"2024-09-12T08:59:34.592310+0000","last_fullsized":"2024-09-12T08:59:34.592310+0000","mapping_epoch":17,"log_start":"162'419236","ondisk_log_start":"162'419236","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:42:22.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930223104,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426680,"num_read_kb":1677524,"num_write":425593,"num_write_kb":1702372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'145","reported_seq":3677,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:32.200034+0000","last_change":"2024-09-12T07:57:31.577407+0000","last_active":"2024-09-12T08:59:32.200034+0000","last_peered":"2024-09-12T08:59:32.200034+0000","last_clean":"2024-09-12T08:59:32.200034+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:32.200034+0000","last_undegraded":"2024-09-12T08:59:32.200034+0000","last_fullsized":"2024-09-12T08:59:32.200034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:26:14.289003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5919,"num_read_kb":4807,"num_write":86,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"165'428859","reported_seq":851077,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.835734+0000","last_change":"2024-09-12T08:50:01.747537+0000","last_active":"2024-09-12T08:59:33.835734+0000","last_peered":"2024-09-12T08:59:33.835734+0000","last_clean":"2024-09-12T08:59:33.835734+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T08:59:33.835734+0000","last_undegraded":"2024-09-12T08:59:33.835734+0000","last_fullsized":"2024-09-12T08:59:33.835734+0000","mapping_epoch":17,"log_start":"162'418842","ondisk_log_start":"162'418842","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:34:43.580846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6927671296,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425473,"num_read_kb":1671764,"num_write":425145,"num_write_kb":1700580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'242","reported_seq":5444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:32.403230+0000","last_change":"2024-09-12T07:57:31.577181+0000","last_active":"2024-09-12T08:59:32.403230+0000","last_peered":"2024-09-12T08:59:32.403230+0000","last_clean":"2024-09-12T08:59:32.403230+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:32.403230+0000","last_undegraded":"2024-09-12T08:59:32.403230+0000","last_fullsized":"2024-09-12T08:59:32.403230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:59:10.221338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8509,"num_read_kb":6945,"num_write":134,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"165'420958","reported_seq":837362,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.600802+0000","last_change":"2024-09-12T08:50:01.692389+0000","last_active":"2024-09-12T08:59:34.600802+0000","last_peered":"2024-09-12T08:59:34.600802+0000","last_clean":"2024-09-12T08:59:34.600802+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T08:59:34.600802+0000","last_undegraded":"2024-09-12T08:59:34.600802+0000","last_fullsized":"2024-09-12T08:59:34.600802+0000","mapping_epoch":17,"log_start":"162'410926","ondisk_log_start":"162'410926","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:58:52.087120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802718739,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419602,"num_read_kb":1648780,"num_write":417305,"num_write_kb":1669214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"165'8830","reported_seq":14099,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.173699+0000","last_change":"2024-09-12T07:57:31.592415+0000","last_active":"2024-09-12T08:59:34.173699+0000","last_peered":"2024-09-12T08:59:34.173699+0000","last_clean":"2024-09-12T08:59:34.173699+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:34.173699+0000","last_undegraded":"2024-09-12T08:59:34.173699+0000","last_fullsized":"2024-09-12T08:59:34.173699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:25:45.764022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8990,"num_read_kb":7314,"num_write":136,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":416,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:13.335552+0000","last_change":"2024-09-12T07:57:26.399925+0000","last_active":"2024-09-12T08:59:13.335552+0000","last_peered":"2024-09-12T08:59:13.335552+0000","last_clean":"2024-09-12T08:59:13.335552+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:13.335552+0000","last_undegraded":"2024-09-12T08:59:13.335552+0000","last_fullsized":"2024-09-12T08:59:13.335552+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_clean_scrub_stamp":"2024-09-12T07:57:24.546601+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:46:35.118087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27647467539,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701174,"num_read_kb":6686228,"num_write":1696426,"num_write_kb":6785698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13894971392,"data_stored":21945020454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40129,"ondisk_log_size":40129,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67570,"num_read_kb":56277,"num_write":1104,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46711,"internal_metadata":0},"log_size":44860,"ondisk_log_size":44860,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706415,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8889128,"kb_used_data":6783832,"kb_used_omap":1495,"kb_used_meta":2103784,"kb_avail":85482712,"statfs":{"total":96636764160,"available":87534297088,"internally_reserved":0,"allocated":6946643968,"data_stored":10970582489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1531196,"internal_metadata":2154275524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706413,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4968264,"kb_used_data":3373432,"kb_used_omap":700,"kb_used_meta":1594115,"kb_avail":89403576,"statfs":{"total":96636764160,"available":91549261824,"internally_reserved":0,"allocated":3454394368,"data_stored":5453281944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1632374214},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5035132,"kb_used_data":3416044,"kb_used_omap":704,"kb_used_meta":1618367,"kb_avail":89336708,"statfs":{"total":96636764160,"available":91480788992,"internally_reserved":0,"allocated":3498029056,"data_stored":5521609838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1657208131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,7,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12848,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3496067072,"data_stored":5520855040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452526592,"data_stored":5452763155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6946377728,"data_stored":10971402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T08:59:37.865 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-12T08:59:37.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T08:59:38.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T08:59:38.166 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T08:59:38.184 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2029,"stamp":"2024-09-12T08:59:36.674964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27648062185,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1768800,"num_read_kb":6742551,"num_write":1697609,"num_write_kb":6787244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85041,"ondisk_log_size":85041,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18892524,"kb_used_data":13573308,"kb_used_omap":2900,"kb_used_meta":5316267,"kb_avail":264222996,"statfs":{"total":289910292480,"available":270564347904,"internally_reserved":0,"allocated":13899067392,"data_stored":21945474271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2970163,"internal_metadata":5443857869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,4,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17747968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5577,"num_read_kb":19016,"num_write":4349,"num_write_kb":17334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004449"},"pg_stats":[{"pgid":"2.7","version":"165'5991","reported_seq":9803,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.075209+0000","last_change":"2024-09-12T07:57:31.593411+0000","last_active":"2024-09-12T08:59:34.075209+0000","last_peered":"2024-09-12T08:59:34.075209+0000","last_clean":"2024-09-12T08:59:34.075209+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:34.075209+0000","last_undegraded":"2024-09-12T08:59:34.075209+0000","last_fullsized":"2024-09-12T08:59:34.075209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:47.434507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6313,"num_read_kb":5114,"num_write":123,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"165'5892","reported_seq":12014,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.406135+0000","last_change":"2024-09-12T07:57:31.592720+0000","last_active":"2024-09-12T08:59:34.406135+0000","last_peered":"2024-09-12T08:59:34.406135+0000","last_clean":"2024-09-12T08:59:34.406135+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:34.406135+0000","last_undegraded":"2024-09-12T08:59:34.406135+0000","last_fullsized":"2024-09-12T08:59:34.406135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:40:49.569427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10704,"num_read_kb":9552,"num_write":126,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"165'8922","reported_seq":14422,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.554788+0000","last_change":"2024-09-12T07:57:31.592903+0000","last_active":"2024-09-12T08:59:34.554788+0000","last_peered":"2024-09-12T08:59:34.554788+0000","last_clean":"2024-09-12T08:59:34.554788+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:34.554788+0000","last_undegraded":"2024-09-12T08:59:34.554788+0000","last_fullsized":"2024-09-12T08:59:34.554788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:31.416374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8864,"num_read_kb":7233,"num_write":172,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'8903","reported_seq":15595,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.267937+0000","last_change":"2024-09-12T07:57:31.592249+0000","last_active":"2024-09-12T08:59:33.267937+0000","last_peered":"2024-09-12T08:59:33.267937+0000","last_clean":"2024-09-12T08:59:33.267937+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:33.267937+0000","last_undegraded":"2024-09-12T08:59:33.267937+0000","last_fullsized":"2024-09-12T08:59:33.267937+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8903,"log_dups_size":0,"ondisk_log_size":8903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:43:47.133587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10611,"num_read_kb":8785,"num_write":168,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"165'432146","reported_seq":858260,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.837025+0000","last_change":"2024-09-12T08:50:01.805419+0000","last_active":"2024-09-12T08:59:33.837025+0000","last_peered":"2024-09-12T08:59:33.837025+0000","last_clean":"2024-09-12T08:59:33.837025+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T08:59:33.837025+0000","last_undegraded":"2024-09-12T08:59:33.837025+0000","last_fullsized":"2024-09-12T08:59:33.837025+0000","mapping_epoch":17,"log_start":"162'422142","ondisk_log_start":"162'422142","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:40:29.170563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6986854400,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429419,"num_read_kb":1688160,"num_write":428383,"num_write_kb":1713532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"165'5935","reported_seq":11177,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.748593+0000","last_change":"2024-09-12T07:57:31.586790+0000","last_active":"2024-09-12T08:59:33.748593+0000","last_peered":"2024-09-12T08:59:33.748593+0000","last_clean":"2024-09-12T08:59:33.748593+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T08:59:33.748593+0000","last_undegraded":"2024-09-12T08:59:33.748593+0000","last_fullsized":"2024-09-12T08:59:33.748593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:26:12.712604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7660,"num_read_kb":6527,"num_write":159,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"165'429312","reported_seq":852731,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.592310+0000","last_change":"2024-09-12T08:50:01.794946+0000","last_active":"2024-09-12T08:59:34.592310+0000","last_peered":"2024-09-12T08:59:34.592310+0000","last_clean":"2024-09-12T08:59:34.592310+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T08:59:34.592310+0000","last_undegraded":"2024-09-12T08:59:34.592310+0000","last_fullsized":"2024-09-12T08:59:34.592310+0000","mapping_epoch":17,"log_start":"162'419236","ondisk_log_start":"162'419236","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:42:22.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930223104,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426680,"num_read_kb":1677524,"num_write":425593,"num_write_kb":1702372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'145","reported_seq":3677,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:32.200034+0000","last_change":"2024-09-12T07:57:31.577407+0000","last_active":"2024-09-12T08:59:32.200034+0000","last_peered":"2024-09-12T08:59:32.200034+0000","last_clean":"2024-09-12T08:59:32.200034+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:32.200034+0000","last_undegraded":"2024-09-12T08:59:32.200034+0000","last_fullsized":"2024-09-12T08:59:32.200034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:26:14.289003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5919,"num_read_kb":4807,"num_write":86,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"165'428859","reported_seq":851077,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:33.835734+0000","last_change":"2024-09-12T08:50:01.747537+0000","last_active":"2024-09-12T08:59:33.835734+0000","last_peered":"2024-09-12T08:59:33.835734+0000","last_clean":"2024-09-12T08:59:33.835734+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T08:59:33.835734+0000","last_undegraded":"2024-09-12T08:59:33.835734+0000","last_fullsized":"2024-09-12T08:59:33.835734+0000","mapping_epoch":17,"log_start":"162'418842","ondisk_log_start":"162'418842","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:34:43.580846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6927671296,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425473,"num_read_kb":1671764,"num_write":425145,"num_write_kb":1700580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'242","reported_seq":5444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:32.403230+0000","last_change":"2024-09-12T07:57:31.577181+0000","last_active":"2024-09-12T08:59:32.403230+0000","last_peered":"2024-09-12T08:59:32.403230+0000","last_clean":"2024-09-12T08:59:32.403230+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:32.403230+0000","last_undegraded":"2024-09-12T08:59:32.403230+0000","last_fullsized":"2024-09-12T08:59:32.403230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:59:10.221338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8509,"num_read_kb":6945,"num_write":134,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"165'420958","reported_seq":837362,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.600802+0000","last_change":"2024-09-12T08:50:01.692389+0000","last_active":"2024-09-12T08:59:34.600802+0000","last_peered":"2024-09-12T08:59:34.600802+0000","last_clean":"2024-09-12T08:59:34.600802+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T08:59:34.600802+0000","last_undegraded":"2024-09-12T08:59:34.600802+0000","last_fullsized":"2024-09-12T08:59:34.600802+0000","mapping_epoch":17,"log_start":"162'410926","ondisk_log_start":"162'410926","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:58:52.087120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802718739,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419602,"num_read_kb":1648780,"num_write":417305,"num_write_kb":1669214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"165'8830","reported_seq":14099,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:34.173699+0000","last_change":"2024-09-12T07:57:31.592415+0000","last_active":"2024-09-12T08:59:34.173699+0000","last_peered":"2024-09-12T08:59:34.173699+0000","last_clean":"2024-09-12T08:59:34.173699+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:34.173699+0000","last_undegraded":"2024-09-12T08:59:34.173699+0000","last_fullsized":"2024-09-12T08:59:34.173699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:25:45.764022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8990,"num_read_kb":7314,"num_write":136,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":416,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:13.335552+0000","last_change":"2024-09-12T07:57:26.399925+0000","last_active":"2024-09-12T08:59:13.335552+0000","last_peered":"2024-09-12T08:59:13.335552+0000","last_clean":"2024-09-12T08:59:13.335552+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:13.335552+0000","last_undegraded":"2024-09-12T08:59:13.335552+0000","last_fullsized":"2024-09-12T08:59:13.335552+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_clean_scrub_stamp":"2024-09-12T07:57:24.546601+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:46:35.118087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27647467539,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701174,"num_read_kb":6686228,"num_write":1696426,"num_write_kb":6785698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13894971392,"data_stored":21945020454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40129,"ondisk_log_size":40129,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67570,"num_read_kb":56277,"num_write":1104,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46711,"internal_metadata":0},"log_size":44860,"ondisk_log_size":44860,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706415,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8889128,"kb_used_data":6783832,"kb_used_omap":1495,"kb_used_meta":2103784,"kb_avail":85482712,"statfs":{"total":96636764160,"available":87534297088,"internally_reserved":0,"allocated":6946643968,"data_stored":10970582489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1531196,"internal_metadata":2154275524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706413,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4968264,"kb_used_data":3373432,"kb_used_omap":700,"kb_used_meta":1594115,"kb_avail":89403576,"statfs":{"total":96636764160,"available":91549261824,"internally_reserved":0,"allocated":3454394368,"data_stored":5453281944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1632374214},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5035132,"kb_used_data":3416044,"kb_used_omap":704,"kb_used_meta":1618367,"kb_avail":89336708,"statfs":{"total":96636764160,"available":91480788992,"internally_reserved":0,"allocated":3498029056,"data_stored":5521609838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1657208131},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,7,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12848,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3496067072,"data_stored":5520855040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452526592,"data_stored":5452763155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6946377728,"data_stored":10971402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T08:59:38.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-12T08:59:38.515 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-12T08:59:38.513+0000 7fabbffff640 0 --2- 172.21.15.137:0/4219445584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabc012b030 0x7fabc0130240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-12T08:59:38.719 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T08:59:38.719 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":166,"fsid":"b2d66243-42d9-4ed0-b04a-17c8c73580cd","created":"2024-09-12T07:57:12.940084+0000","modified":"2024-09-12T08:59:38.325879+0000","last_up_change":"2024-09-12T07:57:23.545318+0000","last_in_change":"2024-09-12T07:57:13.898448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T07:57:23.901045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T07:57:28.240312+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-12T07:57:36.038609+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":101,"snap_epoch":147,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7f68ec7-0a8d-4dca-b021-98e1568bcbc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6801","nonce":998099114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6803","nonce":998099114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6807","nonce":998099114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":998099114},{"type":"v1","addr":"172.21.15.137:6805","nonce":998099114}]},"public_addr":"172.21.15.137:6801/998099114","cluster_addr":"172.21.15.137:6803/998099114","heartbeat_back_addr":"172.21.15.137:6807/998099114","heartbeat_front_addr":"172.21.15.137:6805/998099114","state":["exists","up"]},{"osd":1,"uuid":"b2f260db-e8a7-4a95-98c3-0b180d2c805e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6817","nonce":547903978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6819","nonce":547903978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6823","nonce":547903978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":547903978},{"type":"v1","addr":"172.21.15.137:6821","nonce":547903978}]},"public_addr":"172.21.15.137:6817/547903978","cluster_addr":"172.21.15.137:6819/547903978","heartbeat_back_addr":"172.21.15.137:6823/547903978","heartbeat_front_addr":"172.21.15.137:6821/547903978","state":["exists","up"]},{"osd":2,"uuid":"8a2900b6-32d2-4e1d-99be-dc483c64abfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6809","nonce":99608485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6811","nonce":99608485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6815","nonce":99608485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":99608485},{"type":"v1","addr":"172.21.15.137:6813","nonce":99608485}]},"public_addr":"172.21.15.137:6809/99608485","cluster_addr":"172.21.15.137:6811/99608485","heartbeat_back_addr":"172.21.15.137:6815/99608485","heartbeat_front_addr":"172.21.15.137:6813/99608485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.352359+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:21.139340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T07:57:20.401170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/1860195745":"2024-09-12T09:55:56.146394+0000","172.21.15.137:0/2808455350":"2024-09-12T09:54:47.232564+0000","172.21.15.137:0/1366747705":"2024-09-12T09:53:38.119368+0000","172.21.15.137:0/2267109441":"2024-09-12T09:47:42.022112+0000","172.21.15.137:0/965763560":"2024-09-12T09:45:13.366680+0000","172.21.15.137:0/3407300315":"2024-09-12T09:42:54.979137+0000","172.21.15.137:0/2966137166":"2024-09-12T09:40:36.524333+0000","172.21.15.137:0/4054511077":"2024-09-12T09:39:27.145348+0000","172.21.15.137:0/3456266723":"2024-09-12T09:37:09.000094+0000","172.21.15.137:0/3728958811":"2024-09-12T09:33:42.241323+0000","172.21.15.137:0/2006593960":"2024-09-12T09:32:32.756730+0000","172.21.15.137:0/3301685105":"2024-09-12T09:52:29.020243+0000","172.21.15.137:0/3572572957":"2024-09-12T09:12:25.172621+0000","172.21.15.137:0/4107428414":"2024-09-12T09:49:59.835020+0000","172.21.15.137:0/1133434747":"2024-09-12T09:08:58.855487+0000","172.21.15.137:0/2534148885":"2024-09-12T09:35:59.816371+0000","172.21.15.137:0/3644346538":"2024-09-12T09:13:34.516355+0000","172.21.15.137:0/2900346027":"2024-09-12T09:06:30.603024+0000","172.21.15.137:0/313384079":"2024-09-12T09:51:09.187633+0000","172.21.15.137:0/558705519":"2024-09-12T09:34:50.668362+0000","172.21.15.137:0/2206968216":"2024-09-12T09:20:40.531679+0000","172.21.15.137:0/2340344547":"2024-09-12T09:03:03.018907+0000","172.21.15.137:0/708566858":"2024-09-12T09:04:12.212069+0000","172.21.15.137:0/2085061616":"2024-09-12T09:05:21.383500+0000","172.21.15.137:0/1429115853":"2024-09-12T09:19:31.254877+0000","172.21.15.137:0/3433447522":"2024-09-12T09:01:53.721156+0000","172.21.15.137:0/1041426508":"2024-09-12T09:07:50.248923+0000","172.21.15.137:0/3800000418":"2024-09-12T09:10:07.609938+0000","172.21.15.137:0/2035321561":"2024-09-12T09:00:44.444261+0000","172.21.15.137:0/2613560026":"2024-09-12T09:46:33.019862+0000","172.21.15.137:0/720494988":"2024-09-12T09:38:18.011141+0000","172.21.15.137:0/2294234594":"2024-09-12T09:11:16.372989+0000","172.21.15.137:0/1022528467":"2024-09-12T09:48:51.065135+0000","172.21.15.137:0/1449742175":"2024-09-12T09:24:18.728175+0000","172.21.15.137:0/3549391526":"2024-09-12T09:14:44.451971+0000","172.21.15.137:0/1486950863":"2024-09-12T09:17:13.833772+0000","172.21.15.137:0/2188351739":"2024-09-12T09:21:49.305639+0000","172.21.15.137:0/640503035":"2024-09-12T09:57:05.239848+0000","172.21.15.137:0/2523539757":"2024-09-12T09:26:37.039912+0000","172.21.15.137:0/4117469639":"2024-09-12T09:30:03.953040+0000","172.21.15.137:0/3839580397":"2024-09-12T09:18:22.637756+0000","172.21.15.137:0/4051686922":"2024-09-12T09:22:58.890243+0000","172.21.15.137:0/2446617153":"2024-09-12T09:44:03.981792+0000","172.21.15.137:0/666035049":"2024-09-12T09:25:27.961078+0000","172.21.15.137:0/2595000226":"2024-09-12T09:16:04.656849+0000","172.21.15.137:0/113963503":"2024-09-12T09:28:54.759926+0000","172.21.15.137:0/2111985957":"2024-09-12T09:27:45.842660+0000","172.21.15.137:0/2502491016":"2024-09-12T09:58:14.274683+0000","172.21.15.137:0/1669362779":"2024-09-12T09:41:46.020621+0000","172.21.15.137:0/1096641937":"2024-09-12T09:31:23.899203+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T08:59:39.741 INFO:tasks.ceph:Scrubbing osd.0 2024-09-12T08:59:39.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-12T08:59:40.118 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-12T08:59:40.118 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T08:59:40.118 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-12T08:59:40.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-12T08:59:40.444 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 0 to deep-scrub 2024-09-12T08:59:40.463 INFO:tasks.ceph:Scrubbing osd.1 2024-09-12T08:59:40.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-12T08:59:40.609 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-12T08:59:40.609 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T08:59:40.609 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-12T08:59:40.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-12T08:59:40.917 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 1 to deep-scrub 2024-09-12T08:59:40.936 INFO:tasks.ceph:Scrubbing osd.2 2024-09-12T08:59:40.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-12T08:59:41.084 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-12T08:59:41.084 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T08:59:41.084 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-12T08:59:41.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-12T08:59:41.469 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 2 to deep-scrub 2024-09-12T08:59:41.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T08:59:41.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T08:59:41.848 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T08:59:41.870 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2032,"stamp":"2024-09-12T08:59:40.675756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27649561321,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1769374,"num_read_kb":6744147,"num_write":1698002,"num_write_kb":6788712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85171,"ondisk_log_size":85171,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18971176,"kb_used_data":13581172,"kb_used_omap":2900,"kb_used_meta":5387051,"kb_avail":264144344,"statfs":{"total":289910292480,"available":270483808256,"internally_reserved":0,"allocated":13907120128,"data_stored":21953520402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2970163,"internal_metadata":5516340685},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10063872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3738,"num_read_kb":11012,"num_write":2503,"num_write_kb":9846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004259"},"pg_stats":[{"pgid":"2.7","version":"165'5997","reported_seq":9815,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329278+0000","last_change":"2024-09-12T07:57:31.593411+0000","last_active":"2024-09-12T08:59:38.329278+0000","last_peered":"2024-09-12T08:59:38.329278+0000","last_clean":"2024-09-12T08:59:38.329278+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:38.329278+0000","last_undegraded":"2024-09-12T08:59:38.329278+0000","last_fullsized":"2024-09-12T08:59:38.329278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:36:47.434507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6315,"num_read_kb":5116,"num_write":125,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"165'5898","reported_seq":12027,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329216+0000","last_change":"2024-09-12T07:57:31.592720+0000","last_active":"2024-09-12T08:59:38.329216+0000","last_peered":"2024-09-12T08:59:38.329216+0000","last_clean":"2024-09-12T08:59:38.329216+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:38.329216+0000","last_undegraded":"2024-09-12T08:59:38.329216+0000","last_fullsized":"2024-09-12T08:59:38.329216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:40:49.569427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10707,"num_read_kb":9555,"num_write":128,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"165'8931","reported_seq":14441,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329312+0000","last_change":"2024-09-12T07:57:31.592903+0000","last_active":"2024-09-12T08:59:38.329312+0000","last_peered":"2024-09-12T08:59:38.329312+0000","last_clean":"2024-09-12T08:59:38.329312+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:38.329312+0000","last_undegraded":"2024-09-12T08:59:38.329312+0000","last_fullsized":"2024-09-12T08:59:38.329312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8931,"log_dups_size":0,"ondisk_log_size":8931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:15:31.416374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8870,"num_read_kb":7237,"num_write":175,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'8912","reported_seq":15612,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329160+0000","last_change":"2024-09-12T07:57:31.592249+0000","last_active":"2024-09-12T08:59:38.329160+0000","last_peered":"2024-09-12T08:59:38.329160+0000","last_clean":"2024-09-12T08:59:38.329160+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:38.329160+0000","last_undegraded":"2024-09-12T08:59:38.329160+0000","last_fullsized":"2024-09-12T08:59:38.329160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:43:47.133587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10614,"num_read_kb":8788,"num_write":171,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"165'432277","reported_seq":858521,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:35.051210+0000","last_change":"2024-09-12T08:50:01.805419+0000","last_active":"2024-09-12T08:59:35.051210+0000","last_peered":"2024-09-12T08:59:35.051210+0000","last_clean":"2024-09-12T08:59:35.051210+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T08:59:35.051210+0000","last_undegraded":"2024-09-12T08:59:35.051210+0000","last_fullsized":"2024-09-12T08:59:35.051210+0000","mapping_epoch":17,"log_start":"162'422242","ondisk_log_start":"162'422242","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:40:29.170563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6987386880,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429547,"num_read_kb":1688668,"num_write":428514,"num_write_kb":1714056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"165'5949","reported_seq":11203,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:35.727114+0000","last_change":"2024-09-12T07:57:31.586790+0000","last_active":"2024-09-12T08:59:35.727114+0000","last_peered":"2024-09-12T08:59:35.727114+0000","last_clean":"2024-09-12T08:59:35.727114+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T08:59:35.727114+0000","last_undegraded":"2024-09-12T08:59:35.727114+0000","last_fullsized":"2024-09-12T08:59:35.727114+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":5949,"log_dups_size":0,"ondisk_log_size":5949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:26:12.712604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7671,"num_read_kb":6536,"num_write":167,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"165'429357","reported_seq":852831,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329080+0000","last_change":"2024-09-12T08:50:01.794946+0000","last_active":"2024-09-12T08:59:38.329080+0000","last_peered":"2024-09-12T08:59:38.329080+0000","last_clean":"2024-09-12T08:59:38.329080+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T08:59:38.329080+0000","last_undegraded":"2024-09-12T08:59:38.329080+0000","last_fullsized":"2024-09-12T08:59:38.329080+0000","mapping_epoch":17,"log_start":"162'419336","ondisk_log_start":"162'419336","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:42:22.125410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930407424,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426731,"num_read_kb":1677728,"num_write":425638,"num_write_kb":1702552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"165'149","reported_seq":3696,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:36.068348+0000","last_change":"2024-09-12T07:57:31.577407+0000","last_active":"2024-09-12T08:59:36.068348+0000","last_peered":"2024-09-12T08:59:36.068348+0000","last_clean":"2024-09-12T08:59:36.068348+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:36.068348+0000","last_undegraded":"2024-09-12T08:59:36.068348+0000","last_fullsized":"2024-09-12T08:59:36.068348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:26:14.289003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5969,"num_read_kb":4845,"num_write":88,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"165'428997","reported_seq":851341,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:35.084719+0000","last_change":"2024-09-12T08:50:01.747537+0000","last_active":"2024-09-12T08:59:35.084719+0000","last_peered":"2024-09-12T08:59:35.084719+0000","last_clean":"2024-09-12T08:59:35.084719+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T08:59:35.084719+0000","last_undegraded":"2024-09-12T08:59:35.084719+0000","last_fullsized":"2024-09-12T08:59:35.084719+0000","mapping_epoch":17,"log_start":"162'418942","ondisk_log_start":"162'418942","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:34:43.580846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6928236544,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425599,"num_read_kb":1672268,"num_write":425283,"num_write_kb":1701132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"165'248","reported_seq":5478,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-12T08:59:36.656922+0000","last_change":"2024-09-12T07:57:31.577181+0000","last_active":"2024-09-12T08:59:36.656922+0000","last_peered":"2024-09-12T08:59:36.656922+0000","last_clean":"2024-09-12T08:59:36.656922+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:36.656922+0000","last_undegraded":"2024-09-12T08:59:36.656922+0000","last_fullsized":"2024-09-12T08:59:36.656922+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:59:10.221338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8602,"num_read_kb":7016,"num_write":137,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"165'421011","reported_seq":837474,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329120+0000","last_change":"2024-09-12T08:50:01.692389+0000","last_active":"2024-09-12T08:59:38.329120+0000","last_peered":"2024-09-12T08:59:38.329120+0000","last_clean":"2024-09-12T08:59:38.329120+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T08:59:38.329120+0000","last_undegraded":"2024-09-12T08:59:38.329120+0000","last_fullsized":"2024-09-12T08:59:38.329120+0000","mapping_epoch":17,"log_start":"162'410926","ondisk_log_start":"162'410926","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:58:52.087120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802935827,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419655,"num_read_kb":1648992,"num_write":417358,"num_write_kb":1669426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"165'8839","reported_seq":14126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329191+0000","last_change":"2024-09-12T07:57:31.592415+0000","last_active":"2024-09-12T08:59:38.329191+0000","last_peered":"2024-09-12T08:59:38.329191+0000","last_clean":"2024-09-12T08:59:38.329191+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:38.329191+0000","last_undegraded":"2024-09-12T08:59:38.329191+0000","last_fullsized":"2024-09-12T08:59:38.329191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:28.553868+0000","last_clean_scrub_stamp":"2024-09-12T07:57:28.553868+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:25:45.764022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9038,"num_read_kb":7352,"num_write":139,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":418,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:38.329158+0000","last_change":"2024-09-12T07:57:26.399925+0000","last_active":"2024-09-12T08:59:38.329158+0000","last_peered":"2024-09-12T08:59:38.329158+0000","last_clean":"2024-09-12T08:59:38.329158+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:38.329158+0000","last_undegraded":"2024-09-12T08:59:38.329158+0000","last_fullsized":"2024-09-12T08:59:38.329158+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:24.546601+0000","last_clean_scrub_stamp":"2024-09-12T07:57:24.546601+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:46:35.118087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27648966675,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701532,"num_read_kb":6687656,"num_write":1696793,"num_write_kb":6787166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13900120064,"data_stored":21950169126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67786,"num_read_kb":56445,"num_write":1130,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46711,"internal_metadata":0},"log_size":44923,"ondisk_log_size":44923,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706416,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8938644,"kb_used_data":6788932,"kb_used_omap":1495,"kb_used_meta":2148200,"kb_avail":85433196,"statfs":{"total":96636764160,"available":87483592704,"internally_reserved":0,"allocated":6951866368,"data_stored":10975804889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1531196,"internal_metadata":2199757508},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706414,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4973588,"kb_used_data":3374592,"kb_used_omap":700,"kb_used_meta":1598275,"kb_avail":89398252,"statfs":{"total":96636764160,"available":91543810048,"internally_reserved":0,"allocated":3455582208,"data_stored":5454463179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1636634054},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5058944,"kb_used_data":3417648,"kb_used_omap":704,"kb_used_meta":1640575,"kb_avail":89312896,"statfs":{"total":96636764160,"available":91456405504,"internally_reserved":0,"allocated":3499671552,"data_stored":5523252334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1679949123},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12848,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3497152512,"data_stored":5521940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452907520,"data_stored":5453144083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950060032,"data_stored":10975084563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T08:59:41.874 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.874 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.874 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.874 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.874 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-12T07:57:28.553868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=28, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-12T07:57:24.546601+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=24, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T08:59:41.875 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T09:00:01.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T09:00:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T09:00:02.356 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T09:00:02.388 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2042,"stamp":"2024-09-12T09:00:00.679245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27649561321,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1769374,"num_read_kb":6744147,"num_write":1698002,"num_write_kb":6788712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85263,"ondisk_log_size":85263,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17967164,"kb_used_data":13581196,"kb_used_omap":2890,"kb_used_meta":4383029,"kb_avail":265148356,"statfs":{"total":289910292480,"available":271511916544,"internally_reserved":0,"allocated":13907144704,"data_stored":21953531768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2959782,"internal_metadata":4488222298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7858,"num_omap_keys":134,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002132"},"pg_stats":[{"pgid":"2.7","version":"166'6016","reported_seq":9843,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:44.667699+0000","last_change":"2024-09-12T08:59:44.667699+0000","last_active":"2024-09-12T08:59:44.667699+0000","last_peered":"2024-09-12T08:59:44.667699+0000","last_clean":"2024-09-12T08:59:44.667699+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:44.667699+0000","last_undegraded":"2024-09-12T08:59:44.667699+0000","last_fullsized":"2024-09-12T08:59:44.667699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6016","last_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_deep_scrub":"166'6016","last_deep_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_clean_scrub_stamp":"2024-09-12T08:59:44.667499+0000","objects_scrubbed":21,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:41:17.115397+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6315,"num_read_kb":5116,"num_write":125,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"166'5907","reported_seq":12044,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:45.602355+0000","last_change":"2024-09-12T08:59:45.602355+0000","last_active":"2024-09-12T08:59:45.602355+0000","last_peered":"2024-09-12T08:59:45.602355+0000","last_clean":"2024-09-12T08:59:45.602355+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:45.602355+0000","last_undegraded":"2024-09-12T08:59:45.602355+0000","last_fullsized":"2024-09-12T08:59:45.602355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5907","last_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_deep_scrub":"166'5907","last_deep_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_clean_scrub_stamp":"2024-09-12T08:59:45.602237+0000","objects_scrubbed":11,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:50:43.326131+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10707,"num_read_kb":9555,"num_write":128,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3316,"num_omap_keys":70,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"166'8942","reported_seq":14460,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:46.663069+0000","last_change":"2024-09-12T08:59:46.663069+0000","last_active":"2024-09-12T08:59:46.663069+0000","last_peered":"2024-09-12T08:59:46.663069+0000","last_clean":"2024-09-12T08:59:46.663069+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:46.663069+0000","last_undegraded":"2024-09-12T08:59:46.663069+0000","last_fullsized":"2024-09-12T08:59:46.663069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8942","last_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_deep_scrub":"166'8942","last_deep_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_clean_scrub_stamp":"2024-09-12T08:59:46.662900+0000","objects_scrubbed":14,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:02:29.570202+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8870,"num_read_kb":7237,"num_write":175,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2638,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"166'8925","reported_seq":15633,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.748644+0000","last_change":"2024-09-12T08:59:42.748644+0000","last_active":"2024-09-12T08:59:42.748644+0000","last_peered":"2024-09-12T08:59:42.748644+0000","last_clean":"2024-09-12T08:59:42.748644+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:42.748644+0000","last_undegraded":"2024-09-12T08:59:42.748644+0000","last_fullsized":"2024-09-12T08:59:42.748644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8925","last_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_deep_scrub":"166'8925","last_deep_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.748481+0000","objects_scrubbed":16,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:48:03.460379+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10614,"num_read_kb":8788,"num_write":171,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3384,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"166'433221","reported_seq":859518,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T08:59:58.698643+0000","last_change":"2024-09-12T08:59:43.244683+0000","last_active":"2024-09-12T08:59:58.698643+0000","last_peered":"2024-09-12T08:59:58.698643+0000","last_clean":"2024-09-12T08:59:58.698643+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T08:59:58.698643+0000","last_undegraded":"2024-09-12T08:59:58.698643+0000","last_fullsized":"2024-09-12T08:59:58.698643+0000","mapping_epoch":17,"log_start":"162'423142","ondisk_log_start":"162'423142","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":1134,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6987386880,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429547,"num_read_kb":1688668,"num_write":428514,"num_write_kb":1714056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"166'5964","reported_seq":11228,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.291476+0000","last_change":"2024-09-12T08:59:42.291476+0000","last_active":"2024-09-12T08:59:42.291476+0000","last_peered":"2024-09-12T08:59:42.291476+0000","last_clean":"2024-09-12T08:59:42.291476+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T08:59:42.291476+0000","last_undegraded":"2024-09-12T08:59:42.291476+0000","last_fullsized":"2024-09-12T08:59:42.291476+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5964","last_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_deep_scrub":"166'5964","last_deep_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.291334+0000","objects_scrubbed":17,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:52:22.139064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7671,"num_read_kb":6536,"num_write":167,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1903,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"166'430064","reported_seq":853554,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T08:59:59.750390+0000","last_change":"2024-09-12T08:59:47.554575+0000","last_active":"2024-09-12T08:59:59.750390+0000","last_peered":"2024-09-12T08:59:59.750390+0000","last_clean":"2024-09-12T08:59:59.750390+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T08:59:59.750390+0000","last_undegraded":"2024-09-12T08:59:59.750390+0000","last_fullsized":"2024-09-12T08:59:59.750390+0000","mapping_epoch":17,"log_start":"162'420036","ondisk_log_start":"162'420036","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":833,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930407424,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426731,"num_read_kb":1677728,"num_write":425638,"num_write_kb":1702552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"166'161","reported_seq":3718,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.976412+0000","last_change":"2024-09-12T08:59:42.976412+0000","last_active":"2024-09-12T08:59:42.976412+0000","last_peered":"2024-09-12T08:59:42.976412+0000","last_clean":"2024-09-12T08:59:42.976412+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:42.976412+0000","last_undegraded":"2024-09-12T08:59:42.976412+0000","last_fullsized":"2024-09-12T08:59:42.976412+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'161","last_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_deep_scrub":"166'161","last_deep_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.976148+0000","objects_scrubbed":14,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:11:38.757967+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5969,"num_read_kb":4845,"num_write":88,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"166'430092","reported_seq":852476,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T08:59:58.766119+0000","last_change":"2024-09-12T08:59:41.242611+0000","last_active":"2024-09-12T08:59:58.766119+0000","last_peered":"2024-09-12T08:59:58.766119+0000","last_clean":"2024-09-12T08:59:58.766119+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T08:59:58.766119+0000","last_undegraded":"2024-09-12T08:59:58.766119+0000","last_fullsized":"2024-09-12T08:59:58.766119+0000","mapping_epoch":17,"log_start":"162'420042","ondisk_log_start":"162'420042","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":1328,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 17s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6928236544,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425599,"num_read_kb":1672268,"num_write":425283,"num_write_kb":1701132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"166'259","reported_seq":5500,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.002366+0000","last_change":"2024-09-12T08:59:42.002273+0000","last_active":"2024-09-12T08:59:42.002366+0000","last_peered":"2024-09-12T08:59:42.002366+0000","last_clean":"2024-09-12T08:59:42.002366+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:42.002366+0000","last_undegraded":"2024-09-12T08:59:42.002366+0000","last_fullsized":"2024-09-12T08:59:42.002366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'259","last_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_deep_scrub":"166'259","last_deep_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.002115+0000","objects_scrubbed":14,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:48:42.106429+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8602,"num_read_kb":7016,"num_write":137,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2639,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"166'421652","reported_seq":838150,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T08:59:59.875914+0000","last_change":"2024-09-12T08:59:48.601948+0000","last_active":"2024-09-12T08:59:59.875914+0000","last_peered":"2024-09-12T08:59:59.875914+0000","last_clean":"2024-09-12T08:59:59.875914+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T08:59:59.875914+0000","last_undegraded":"2024-09-12T08:59:59.875914+0000","last_fullsized":"2024-09-12T08:59:59.875914+0000","mapping_epoch":17,"log_start":"162'411626","ondisk_log_start":"162'411626","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":757,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802935827,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419655,"num_read_kb":1648992,"num_write":417358,"num_write_kb":1669426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"166'8852","reported_seq":14147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:41.685485+0000","last_change":"2024-09-12T08:59:41.685485+0000","last_active":"2024-09-12T08:59:41.685485+0000","last_peered":"2024-09-12T08:59:41.685485+0000","last_clean":"2024-09-12T08:59:41.685485+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:41.685485+0000","last_undegraded":"2024-09-12T08:59:41.685485+0000","last_fullsized":"2024-09-12T08:59:41.685485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8852","last_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_deep_scrub":"166'8852","last_deep_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_clean_scrub_stamp":"2024-09-12T08:59:41.685356+0000","objects_scrubbed":16,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:22.359185+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9038,"num_read_kb":7352,"num_write":139,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"166'54","reported_seq":429,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:43.643601+0000","last_change":"2024-09-12T08:59:43.643514+0000","last_active":"2024-09-12T08:59:43.643601+0000","last_peered":"2024-09-12T08:59:43.643601+0000","last_clean":"2024-09-12T08:59:43.643601+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:43.643601+0000","last_undegraded":"2024-09-12T08:59:43.643601+0000","last_fullsized":"2024-09-12T08:59:43.643601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'54","last_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_deep_scrub":"166'54","last_deep_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_clean_scrub_stamp":"2024-09-12T08:59:43.643403+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:29:42.340048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27648966675,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701532,"num_read_kb":6687656,"num_write":1696793,"num_write_kb":6787166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13900120064,"data_stored":21950169126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40183,"ondisk_log_size":40183,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67786,"num_read_kb":56445,"num_write":1130,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44633,"internal_metadata":0},"log_size":45026,"ondisk_log_size":45026,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706420,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7955424,"kb_used_data":6788944,"kb_used_omap":1485,"kb_used_meta":1164978,"kb_avail":86416416,"statfs":{"total":96636764160,"available":88490409984,"internally_reserved":0,"allocated":6951878656,"data_stored":10975810572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1520815,"internal_metadata":1192938321},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706418,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4961104,"kb_used_data":3374592,"kb_used_omap":700,"kb_used_meta":1585795,"kb_avail":89410736,"statfs":{"total":96636764160,"available":91556593664,"internally_reserved":0,"allocated":3455582208,"data_stored":5454463179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1623854534},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5050636,"kb_used_data":3417660,"kb_used_omap":704,"kb_used_meta":1632255,"kb_avail":89321204,"statfs":{"total":96636764160,"available":91464912896,"internally_reserved":0,"allocated":3499683840,"data_stored":5523258017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1671429443},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10770,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3497152512,"data_stored":5521940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452907520,"data_stored":5453144083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950060032,"data_stored":10975084563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T09:00:02.390 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:02.390 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:02.390 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:02.390 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:02.390 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T09:00:18.111 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-12T09:00:18.104+0000 7f9d2e42d640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-12T09:00:22.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T09:00:22.917 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T09:00:22.917 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T09:00:22.950 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2053,"stamp":"2024-09-12T09:00:22.683137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27649561321,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1769374,"num_read_kb":6744147,"num_write":1698002,"num_write_kb":6788712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85309,"ondisk_log_size":85309,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18016508,"kb_used_data":13581196,"kb_used_omap":2890,"kb_used_meta":4432373,"kb_avail":265099012,"statfs":{"total":289910292480,"available":271461388288,"internally_reserved":0,"allocated":13907144704,"data_stored":21953531768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2959782,"internal_metadata":4538750554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,4,3,6,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002159"},"pg_stats":[{"pgid":"2.7","version":"166'6016","reported_seq":9843,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:44.667699+0000","last_change":"2024-09-12T08:59:44.667699+0000","last_active":"2024-09-12T08:59:44.667699+0000","last_peered":"2024-09-12T08:59:44.667699+0000","last_clean":"2024-09-12T08:59:44.667699+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:44.667699+0000","last_undegraded":"2024-09-12T08:59:44.667699+0000","last_fullsized":"2024-09-12T08:59:44.667699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6016","last_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_deep_scrub":"166'6016","last_deep_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_clean_scrub_stamp":"2024-09-12T08:59:44.667499+0000","objects_scrubbed":21,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:41:17.115397+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6315,"num_read_kb":5116,"num_write":125,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"166'5907","reported_seq":12044,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:45.602355+0000","last_change":"2024-09-12T08:59:45.602355+0000","last_active":"2024-09-12T08:59:45.602355+0000","last_peered":"2024-09-12T08:59:45.602355+0000","last_clean":"2024-09-12T08:59:45.602355+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:45.602355+0000","last_undegraded":"2024-09-12T08:59:45.602355+0000","last_fullsized":"2024-09-12T08:59:45.602355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5907","last_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_deep_scrub":"166'5907","last_deep_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_clean_scrub_stamp":"2024-09-12T08:59:45.602237+0000","objects_scrubbed":11,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:50:43.326131+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10707,"num_read_kb":9555,"num_write":128,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3316,"num_omap_keys":70,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"166'8942","reported_seq":14461,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:46.663164+0000","last_change":"2024-09-12T08:59:46.663069+0000","last_active":"2024-09-12T08:59:46.663164+0000","last_peered":"2024-09-12T08:59:46.663164+0000","last_clean":"2024-09-12T08:59:46.663164+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:46.663164+0000","last_undegraded":"2024-09-12T08:59:46.663164+0000","last_fullsized":"2024-09-12T08:59:46.663164+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8942","last_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_deep_scrub":"166'8942","last_deep_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_clean_scrub_stamp":"2024-09-12T08:59:46.662900+0000","objects_scrubbed":14,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:02:29.570202+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8870,"num_read_kb":7237,"num_write":175,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2638,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"166'8925","reported_seq":15634,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.748707+0000","last_change":"2024-09-12T08:59:42.748644+0000","last_active":"2024-09-12T08:59:42.748707+0000","last_peered":"2024-09-12T08:59:42.748707+0000","last_clean":"2024-09-12T08:59:42.748707+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:42.748707+0000","last_undegraded":"2024-09-12T08:59:42.748707+0000","last_fullsized":"2024-09-12T08:59:42.748707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8925","last_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_deep_scrub":"166'8925","last_deep_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.748481+0000","objects_scrubbed":16,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:48:03.460379+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10614,"num_read_kb":8788,"num_write":171,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3384,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"166'434474","reported_seq":860824,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:19.194449+0000","last_change":"2024-09-12T08:59:43.244683+0000","last_active":"2024-09-12T09:00:19.194449+0000","last_peered":"2024-09-12T09:00:19.194449+0000","last_clean":"2024-09-12T09:00:19.194449+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T09:00:19.194449+0000","last_undegraded":"2024-09-12T09:00:19.194449+0000","last_fullsized":"2024-09-12T09:00:19.194449+0000","mapping_epoch":17,"log_start":"164'424442","ondisk_log_start":"164'424442","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":2413,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 35s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6987386880,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429547,"num_read_kb":1688668,"num_write":428514,"num_write_kb":1714056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"166'5964","reported_seq":11229,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.291530+0000","last_change":"2024-09-12T08:59:42.291476+0000","last_active":"2024-09-12T08:59:42.291530+0000","last_peered":"2024-09-12T08:59:42.291530+0000","last_clean":"2024-09-12T08:59:42.291530+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T08:59:42.291530+0000","last_undegraded":"2024-09-12T08:59:42.291530+0000","last_fullsized":"2024-09-12T08:59:42.291530+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5964","last_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_deep_scrub":"166'5964","last_deep_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.291334+0000","objects_scrubbed":17,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:52:22.139064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7671,"num_read_kb":6536,"num_write":167,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1903,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"166'431299","reported_seq":854864,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:20.054169+0000","last_change":"2024-09-12T08:59:47.554575+0000","last_active":"2024-09-12T09:00:20.054169+0000","last_peered":"2024-09-12T09:00:20.054169+0000","last_clean":"2024-09-12T09:00:20.054169+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T09:00:20.054169+0000","last_undegraded":"2024-09-12T09:00:20.054169+0000","last_fullsized":"2024-09-12T09:00:20.054169+0000","mapping_epoch":17,"log_start":"164'421236","ondisk_log_start":"164'421236","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":2085,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 32s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930407424,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426731,"num_read_kb":1677728,"num_write":425638,"num_write_kb":1702552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"166'161","reported_seq":3719,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.976560+0000","last_change":"2024-09-12T08:59:42.976412+0000","last_active":"2024-09-12T08:59:42.976560+0000","last_peered":"2024-09-12T08:59:42.976560+0000","last_clean":"2024-09-12T08:59:42.976560+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:42.976560+0000","last_undegraded":"2024-09-12T08:59:42.976560+0000","last_fullsized":"2024-09-12T08:59:42.976560+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'161","last_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_deep_scrub":"166'161","last_deep_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.976148+0000","objects_scrubbed":14,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:11:38.757967+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5969,"num_read_kb":4845,"num_write":88,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"166'431308","reported_seq":853744,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:19.352298+0000","last_change":"2024-09-12T08:59:41.242611+0000","last_active":"2024-09-12T09:00:19.352298+0000","last_peered":"2024-09-12T09:00:19.352298+0000","last_clean":"2024-09-12T09:00:19.352298+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T09:00:19.352298+0000","last_undegraded":"2024-09-12T09:00:19.352298+0000","last_fullsized":"2024-09-12T09:00:19.352298+0000","mapping_epoch":17,"log_start":"164'421242","ondisk_log_start":"164'421242","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":2564,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 38s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6928236544,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425599,"num_read_kb":1672268,"num_write":425283,"num_write_kb":1701132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"166'259","reported_seq":5500,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.002366+0000","last_change":"2024-09-12T08:59:42.002273+0000","last_active":"2024-09-12T08:59:42.002366+0000","last_peered":"2024-09-12T08:59:42.002366+0000","last_clean":"2024-09-12T08:59:42.002366+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:42.002366+0000","last_undegraded":"2024-09-12T08:59:42.002366+0000","last_fullsized":"2024-09-12T08:59:42.002366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'259","last_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_deep_scrub":"166'259","last_deep_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.002115+0000","objects_scrubbed":14,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:48:42.106429+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8602,"num_read_kb":7016,"num_write":137,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2639,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"166'422894","reported_seq":839441,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:20.100568+0000","last_change":"2024-09-12T08:59:48.601948+0000","last_active":"2024-09-12T09:00:20.100568+0000","last_peered":"2024-09-12T09:00:20.100568+0000","last_clean":"2024-09-12T09:00:20.100568+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T09:00:20.100568+0000","last_undegraded":"2024-09-12T09:00:20.100568+0000","last_fullsized":"2024-09-12T09:00:20.100568+0000","mapping_epoch":17,"log_start":"164'412826","ondisk_log_start":"164'412826","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":2016,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 31s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802935827,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419655,"num_read_kb":1648992,"num_write":417358,"num_write_kb":1669426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"166'8852","reported_seq":14148,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:41.685565+0000","last_change":"2024-09-12T08:59:41.685485+0000","last_active":"2024-09-12T08:59:41.685565+0000","last_peered":"2024-09-12T08:59:41.685565+0000","last_clean":"2024-09-12T08:59:41.685565+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:41.685565+0000","last_undegraded":"2024-09-12T08:59:41.685565+0000","last_fullsized":"2024-09-12T08:59:41.685565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8852","last_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_deep_scrub":"166'8852","last_deep_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_clean_scrub_stamp":"2024-09-12T08:59:41.685356+0000","objects_scrubbed":16,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:22.359185+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9038,"num_read_kb":7352,"num_write":139,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"166'54","reported_seq":429,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:43.643601+0000","last_change":"2024-09-12T08:59:43.643514+0000","last_active":"2024-09-12T08:59:43.643601+0000","last_peered":"2024-09-12T08:59:43.643601+0000","last_clean":"2024-09-12T08:59:43.643601+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:43.643601+0000","last_undegraded":"2024-09-12T08:59:43.643601+0000","last_fullsized":"2024-09-12T08:59:43.643601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'54","last_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_deep_scrub":"166'54","last_deep_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_clean_scrub_stamp":"2024-09-12T08:59:43.643403+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:29:42.340048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27648966675,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701532,"num_read_kb":6687656,"num_write":1696793,"num_write_kb":6787166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13900120064,"data_stored":21950169126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40229,"ondisk_log_size":40229,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67786,"num_read_kb":56445,"num_write":1130,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44633,"internal_metadata":0},"log_size":45026,"ondisk_log_size":45026,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706424,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7959584,"kb_used_data":6788944,"kb_used_omap":1485,"kb_used_meta":1169138,"kb_avail":86412256,"statfs":{"total":96636764160,"available":88486150144,"internally_reserved":0,"allocated":6951878656,"data_stored":10975810572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1520815,"internal_metadata":1197198161},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706422,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4997968,"kb_used_data":3374592,"kb_used_omap":700,"kb_used_meta":1622659,"kb_avail":89373872,"statfs":{"total":96636764160,"available":91518844928,"internally_reserved":0,"allocated":3455582208,"data_stored":5454463179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1661603270},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5058956,"kb_used_data":3417660,"kb_used_omap":704,"kb_used_meta":1640575,"kb_avail":89312884,"statfs":{"total":96636764160,"available":91456393216,"internally_reserved":0,"allocated":3499683840,"data_stored":5523258017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1679949123},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,6,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10770,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3497152512,"data_stored":5521940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452907520,"data_stored":5453144083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950060032,"data_stored":10975084563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T09:00:22.952 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:22.952 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:22.952 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:22.952 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:22.952 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T09:00:42.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T09:00:43.432 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T09:00:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T09:00:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2063,"stamp":"2024-09-12T09:00:42.686722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27649561321,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1769375,"num_read_kb":6744148,"num_write":1698002,"num_write_kb":6788712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85267,"ondisk_log_size":85267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18080316,"kb_used_data":13581196,"kb_used_omap":2890,"kb_used_meta":4496181,"kb_avail":265035204,"statfs":{"total":289910292480,"available":271396048896,"internally_reserved":0,"allocated":13907144704,"data_stored":21953531768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2959782,"internal_metadata":4604089946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,11,4,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002229"},"pg_stats":[{"pgid":"2.7","version":"166'6016","reported_seq":9843,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:44.667699+0000","last_change":"2024-09-12T08:59:44.667699+0000","last_active":"2024-09-12T08:59:44.667699+0000","last_peered":"2024-09-12T08:59:44.667699+0000","last_clean":"2024-09-12T08:59:44.667699+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T08:59:44.667699+0000","last_undegraded":"2024-09-12T08:59:44.667699+0000","last_fullsized":"2024-09-12T08:59:44.667699+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6016","last_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_deep_scrub":"166'6016","last_deep_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_clean_scrub_stamp":"2024-09-12T08:59:44.667499+0000","objects_scrubbed":21,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:41:17.115397+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6315,"num_read_kb":5116,"num_write":125,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"166'5907","reported_seq":12045,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:45.602413+0000","last_change":"2024-09-12T08:59:45.602355+0000","last_active":"2024-09-12T08:59:45.602413+0000","last_peered":"2024-09-12T08:59:45.602413+0000","last_clean":"2024-09-12T08:59:45.602413+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T08:59:45.602413+0000","last_undegraded":"2024-09-12T08:59:45.602413+0000","last_fullsized":"2024-09-12T08:59:45.602413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5907","last_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_deep_scrub":"166'5907","last_deep_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_clean_scrub_stamp":"2024-09-12T08:59:45.602237+0000","objects_scrubbed":11,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:50:43.326131+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10707,"num_read_kb":9555,"num_write":128,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3316,"num_omap_keys":70,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"166'8942","reported_seq":14461,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:46.663164+0000","last_change":"2024-09-12T08:59:46.663069+0000","last_active":"2024-09-12T08:59:46.663164+0000","last_peered":"2024-09-12T08:59:46.663164+0000","last_clean":"2024-09-12T08:59:46.663164+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T08:59:46.663164+0000","last_undegraded":"2024-09-12T08:59:46.663164+0000","last_fullsized":"2024-09-12T08:59:46.663164+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8942","last_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_deep_scrub":"166'8942","last_deep_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_clean_scrub_stamp":"2024-09-12T08:59:46.662900+0000","objects_scrubbed":14,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:02:29.570202+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8870,"num_read_kb":7237,"num_write":175,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2638,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"166'8925","reported_seq":15634,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.748707+0000","last_change":"2024-09-12T08:59:42.748644+0000","last_active":"2024-09-12T08:59:42.748707+0000","last_peered":"2024-09-12T08:59:42.748707+0000","last_clean":"2024-09-12T08:59:42.748707+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T08:59:42.748707+0000","last_undegraded":"2024-09-12T08:59:42.748707+0000","last_fullsized":"2024-09-12T08:59:42.748707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8925","last_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_deep_scrub":"166'8925","last_deep_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.748481+0000","objects_scrubbed":16,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:48:03.460379+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10614,"num_read_kb":8788,"num_write":171,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3384,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"166'435728","reported_seq":862130,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:39.460797+0000","last_change":"2024-09-12T08:59:43.244683+0000","last_active":"2024-09-12T09:00:39.460797+0000","last_peered":"2024-09-12T09:00:39.460797+0000","last_clean":"2024-09-12T09:00:39.460797+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T09:00:39.460797+0000","last_undegraded":"2024-09-12T09:00:39.460797+0000","last_fullsized":"2024-09-12T09:00:39.460797+0000","mapping_epoch":17,"log_start":"164'425642","ondisk_log_start":"164'425642","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":3667,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6987386880,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429547,"num_read_kb":1688668,"num_write":428514,"num_write_kb":1714056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"166'5964","reported_seq":11230,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T09:00:19.352501+0000","last_change":"2024-09-12T08:59:42.291476+0000","last_active":"2024-09-12T09:00:19.352501+0000","last_peered":"2024-09-12T09:00:19.352501+0000","last_clean":"2024-09-12T09:00:19.352501+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T09:00:19.352501+0000","last_undegraded":"2024-09-12T09:00:19.352501+0000","last_fullsized":"2024-09-12T09:00:19.352501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5964","last_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_deep_scrub":"166'5964","last_deep_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.291334+0000","objects_scrubbed":17,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:52:22.139064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7672,"num_read_kb":6537,"num_write":167,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1903,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"166'432579","reported_seq":856186,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:40.305754+0000","last_change":"2024-09-12T08:59:47.554575+0000","last_active":"2024-09-12T09:00:40.305754+0000","last_peered":"2024-09-12T09:00:40.305754+0000","last_clean":"2024-09-12T09:00:40.305754+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T09:00:40.305754+0000","last_undegraded":"2024-09-12T09:00:40.305754+0000","last_fullsized":"2024-09-12T09:00:40.305754+0000","mapping_epoch":17,"log_start":"164'422536","ondisk_log_start":"164'422536","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":3365,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 52s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930407424,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426731,"num_read_kb":1677728,"num_write":425638,"num_write_kb":1702552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"166'161","reported_seq":3719,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.976560+0000","last_change":"2024-09-12T08:59:42.976412+0000","last_active":"2024-09-12T08:59:42.976560+0000","last_peered":"2024-09-12T08:59:42.976560+0000","last_clean":"2024-09-12T08:59:42.976560+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T08:59:42.976560+0000","last_undegraded":"2024-09-12T08:59:42.976560+0000","last_fullsized":"2024-09-12T08:59:42.976560+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'161","last_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_deep_scrub":"166'161","last_deep_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.976148+0000","objects_scrubbed":14,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:11:38.757967+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5969,"num_read_kb":4845,"num_write":88,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"166'432557","reported_seq":855065,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:39.454863+0000","last_change":"2024-09-12T08:59:41.242611+0000","last_active":"2024-09-12T09:00:39.454863+0000","last_peered":"2024-09-12T09:00:39.454863+0000","last_clean":"2024-09-12T09:00:39.454863+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T09:00:39.454863+0000","last_undegraded":"2024-09-12T09:00:39.454863+0000","last_fullsized":"2024-09-12T09:00:39.454863+0000","mapping_epoch":17,"log_start":"164'422542","ondisk_log_start":"164'422542","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":3813,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 58s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6928236544,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425599,"num_read_kb":1672268,"num_write":425283,"num_write_kb":1701132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"166'259","reported_seq":5500,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:42.002366+0000","last_change":"2024-09-12T08:59:42.002273+0000","last_active":"2024-09-12T08:59:42.002366+0000","last_peered":"2024-09-12T08:59:42.002366+0000","last_clean":"2024-09-12T08:59:42.002366+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T08:59:42.002366+0000","last_undegraded":"2024-09-12T08:59:42.002366+0000","last_fullsized":"2024-09-12T08:59:42.002366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'259","last_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_deep_scrub":"166'259","last_deep_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.002115+0000","objects_scrubbed":14,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:48:42.106429+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8602,"num_read_kb":7016,"num_write":137,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2639,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"166'424169","reported_seq":840773,"reported_epoch":166,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:40.194499+0000","last_change":"2024-09-12T08:59:48.601948+0000","last_active":"2024-09-12T09:00:40.194499+0000","last_peered":"2024-09-12T09:00:40.194499+0000","last_clean":"2024-09-12T09:00:40.194499+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T09:00:40.194499+0000","last_undegraded":"2024-09-12T09:00:40.194499+0000","last_fullsized":"2024-09-12T09:00:40.194499+0000","mapping_epoch":17,"log_start":"164'414126","ondisk_log_start":"164'414126","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":3291,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 51s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802935827,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419655,"num_read_kb":1648992,"num_write":417358,"num_write_kb":1669426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"166'8852","reported_seq":14148,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:41.685565+0000","last_change":"2024-09-12T08:59:41.685485+0000","last_active":"2024-09-12T08:59:41.685565+0000","last_peered":"2024-09-12T08:59:41.685565+0000","last_clean":"2024-09-12T08:59:41.685565+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T08:59:41.685565+0000","last_undegraded":"2024-09-12T08:59:41.685565+0000","last_fullsized":"2024-09-12T08:59:41.685565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8852","last_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_deep_scrub":"166'8852","last_deep_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_clean_scrub_stamp":"2024-09-12T08:59:41.685356+0000","objects_scrubbed":16,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:22.359185+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9038,"num_read_kb":7352,"num_write":139,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"166'54","reported_seq":429,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-12T08:59:43.643601+0000","last_change":"2024-09-12T08:59:43.643514+0000","last_active":"2024-09-12T08:59:43.643601+0000","last_peered":"2024-09-12T08:59:43.643601+0000","last_clean":"2024-09-12T08:59:43.643601+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T08:59:43.643601+0000","last_undegraded":"2024-09-12T08:59:43.643601+0000","last_fullsized":"2024-09-12T08:59:43.643601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'54","last_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_deep_scrub":"166'54","last_deep_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_clean_scrub_stamp":"2024-09-12T08:59:43.643403+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:29:42.340048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27648966675,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701532,"num_read_kb":6687656,"num_write":1696793,"num_write_kb":6787166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13900120064,"data_stored":21950169126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40187,"ondisk_log_size":40187,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67787,"num_read_kb":56446,"num_write":1130,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44633,"internal_metadata":0},"log_size":45026,"ondisk_log_size":45026,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706428,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7999840,"kb_used_data":6788944,"kb_used_omap":1485,"kb_used_meta":1209394,"kb_avail":86372000,"statfs":{"total":96636764160,"available":88444928000,"internally_reserved":0,"allocated":6951878656,"data_stored":10975810572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1520815,"internal_metadata":1238420305},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706426,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5007696,"kb_used_data":3374592,"kb_used_omap":700,"kb_used_meta":1632387,"kb_avail":89364144,"statfs":{"total":96636764160,"available":91508883456,"internally_reserved":0,"allocated":3455582208,"data_stored":5454463179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1671564742},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072780,"kb_used_data":3417660,"kb_used_omap":704,"kb_used_meta":1654399,"kb_avail":89299060,"statfs":{"total":96636764160,"available":91442237440,"internally_reserved":0,"allocated":3499683840,"data_stored":5523258017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1694104899},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,11,4,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10770,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3497152512,"data_stored":5521940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452907520,"data_stored":5453144083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950060032,"data_stored":10975084563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T09:00:43.459 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:43.459 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:43.459 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:43.459 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:00:43.459 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T09:01:03.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T09:01:03.872 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T09:01:03.872 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T09:01:03.901 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2074,"stamp":"2024-09-12T09:01:02.690170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27649561321,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1769375,"num_read_kb":6744148,"num_write":1698002,"num_write_kb":6788712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85335,"ondisk_log_size":85335,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18160864,"kb_used_data":13581232,"kb_used_omap":2890,"kb_used_meta":4576693,"kb_avail":264954656,"statfs":{"total":289910292480,"available":271313567744,"internally_reserved":0,"allocated":13907181568,"data_stored":21953548688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2959782,"internal_metadata":4686534234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,0,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002173"},"pg_stats":[{"pgid":"2.7","version":"166'6016","reported_seq":9845,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.518028+0000","last_change":"2024-09-12T08:59:44.667699+0000","last_active":"2024-09-12T09:00:48.518028+0000","last_peered":"2024-09-12T09:00:48.518028+0000","last_clean":"2024-09-12T09:00:48.518028+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T09:00:48.518028+0000","last_undegraded":"2024-09-12T09:00:48.518028+0000","last_fullsized":"2024-09-12T09:00:48.518028+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6016","last_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_deep_scrub":"166'6016","last_deep_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_clean_scrub_stamp":"2024-09-12T08:59:44.667499+0000","objects_scrubbed":21,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:41:17.115397+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6315,"num_read_kb":5116,"num_write":125,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"166'5907","reported_seq":12047,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517943+0000","last_change":"2024-09-12T08:59:45.602355+0000","last_active":"2024-09-12T09:00:48.517943+0000","last_peered":"2024-09-12T09:00:48.517943+0000","last_clean":"2024-09-12T09:00:48.517943+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T09:00:48.517943+0000","last_undegraded":"2024-09-12T09:00:48.517943+0000","last_fullsized":"2024-09-12T09:00:48.517943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5907","last_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_deep_scrub":"166'5907","last_deep_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_clean_scrub_stamp":"2024-09-12T08:59:45.602237+0000","objects_scrubbed":11,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:50:43.326131+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10707,"num_read_kb":9555,"num_write":128,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3316,"num_omap_keys":70,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"166'8942","reported_seq":14463,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517929+0000","last_change":"2024-09-12T08:59:46.663069+0000","last_active":"2024-09-12T09:00:48.517929+0000","last_peered":"2024-09-12T09:00:48.517929+0000","last_clean":"2024-09-12T09:00:48.517929+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T09:00:48.517929+0000","last_undegraded":"2024-09-12T09:00:48.517929+0000","last_fullsized":"2024-09-12T09:00:48.517929+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8942","last_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_deep_scrub":"166'8942","last_deep_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_clean_scrub_stamp":"2024-09-12T08:59:46.662900+0000","objects_scrubbed":14,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:02:29.570202+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8870,"num_read_kb":7237,"num_write":175,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2638,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"166'8925","reported_seq":15636,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517921+0000","last_change":"2024-09-12T08:59:42.748644+0000","last_active":"2024-09-12T09:00:48.517921+0000","last_peered":"2024-09-12T09:00:48.517921+0000","last_clean":"2024-09-12T09:00:48.517921+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T09:00:48.517921+0000","last_undegraded":"2024-09-12T09:00:48.517921+0000","last_fullsized":"2024-09-12T09:00:48.517921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8925","last_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_deep_scrub":"166'8925","last_deep_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.748481+0000","objects_scrubbed":16,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:48:03.460379+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10614,"num_read_kb":8788,"num_write":171,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3384,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"167'437004","reported_seq":863438,"reported_epoch":167,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:00:59.602642+0000","last_change":"2024-09-12T08:59:43.244683+0000","last_active":"2024-09-12T09:00:59.602642+0000","last_peered":"2024-09-12T09:00:59.602642+0000","last_clean":"2024-09-12T09:00:59.602642+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T09:00:59.602642+0000","last_undegraded":"2024-09-12T09:00:59.602642+0000","last_fullsized":"2024-09-12T09:00:59.602642+0000","mapping_epoch":17,"log_start":"164'426942","ondisk_log_start":"164'426942","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":4943,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 76s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6987386880,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429547,"num_read_kb":1688668,"num_write":428514,"num_write_kb":1714056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"166'5964","reported_seq":11232,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.343361+0000","last_change":"2024-09-12T08:59:42.291476+0000","last_active":"2024-09-12T09:00:48.343361+0000","last_peered":"2024-09-12T09:00:48.343361+0000","last_clean":"2024-09-12T09:00:48.343361+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T09:00:48.343361+0000","last_undegraded":"2024-09-12T09:00:48.343361+0000","last_fullsized":"2024-09-12T09:00:48.343361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5964","last_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_deep_scrub":"166'5964","last_deep_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.291334+0000","objects_scrubbed":17,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:52:22.139064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7672,"num_read_kb":6537,"num_write":167,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1903,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"167'433880","reported_seq":857533,"reported_epoch":167,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:01:00.617222+0000","last_change":"2024-09-12T08:59:47.554575+0000","last_active":"2024-09-12T09:01:00.617222+0000","last_peered":"2024-09-12T09:01:00.617222+0000","last_clean":"2024-09-12T09:01:00.617222+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T09:01:00.617222+0000","last_undegraded":"2024-09-12T09:01:00.617222+0000","last_fullsized":"2024-09-12T09:01:00.617222+0000","mapping_epoch":17,"log_start":"164'423836","ondisk_log_start":"164'423836","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":4666,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 73s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930407424,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426731,"num_read_kb":1677728,"num_write":425638,"num_write_kb":1702552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"166'161","reported_seq":3721,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.409802+0000","last_change":"2024-09-12T08:59:42.976412+0000","last_active":"2024-09-12T09:00:48.409802+0000","last_peered":"2024-09-12T09:00:48.409802+0000","last_clean":"2024-09-12T09:00:48.409802+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T09:00:48.409802+0000","last_undegraded":"2024-09-12T09:00:48.409802+0000","last_fullsized":"2024-09-12T09:00:48.409802+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'161","last_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_deep_scrub":"166'161","last_deep_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.976148+0000","objects_scrubbed":14,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:11:38.757967+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5969,"num_read_kb":4845,"num_write":88,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"167'433741","reported_seq":856305,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:58.110106+0000","last_change":"2024-09-12T09:00:58.110106+0000","last_active":"2024-09-12T09:00:58.110106+0000","last_peered":"2024-09-12T09:00:58.110106+0000","last_clean":"2024-09-12T09:00:58.110106+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T09:00:58.110106+0000","last_undegraded":"2024-09-12T09:00:58.110106+0000","last_fullsized":"2024-09-12T09:00:58.110106+0000","mapping_epoch":17,"log_start":"164'423642","ondisk_log_start":"164'423642","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'433741","last_scrub_stamp":"2024-09-12T09:00:58.109963+0000","last_deep_scrub":"167'433741","last_deep_scrub_stamp":"2024-09-12T09:00:58.109963+0000","last_clean_scrub_stamp":"2024-09-12T09:00:58.109963+0000","objects_scrubbed":4997,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":77,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:45:16.158082+0000","scrub_duration":76862,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6928236544,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425599,"num_read_kb":1672268,"num_write":425283,"num_write_kb":1701132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"166'259","reported_seq":5502,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.410743+0000","last_change":"2024-09-12T08:59:42.002273+0000","last_active":"2024-09-12T09:00:48.410743+0000","last_peered":"2024-09-12T09:00:48.410743+0000","last_clean":"2024-09-12T09:00:48.410743+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T09:00:48.410743+0000","last_undegraded":"2024-09-12T09:00:48.410743+0000","last_fullsized":"2024-09-12T09:00:48.410743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'259","last_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_deep_scrub":"166'259","last_deep_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.002115+0000","objects_scrubbed":14,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:48:42.106429+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8602,"num_read_kb":7016,"num_write":137,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2639,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"167'425476","reported_seq":842136,"reported_epoch":167,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-12T09:01:00.488261+0000","last_change":"2024-09-12T08:59:48.601948+0000","last_active":"2024-09-12T09:01:00.488261+0000","last_peered":"2024-09-12T09:01:00.488261+0000","last_clean":"2024-09-12T09:01:00.488261+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T09:01:00.488261+0000","last_undegraded":"2024-09-12T09:01:00.488261+0000","last_fullsized":"2024-09-12T09:01:00.488261+0000","mapping_epoch":17,"log_start":"164'415426","ondisk_log_start":"164'415426","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T07:57:36.587959+0000","last_clean_scrub_stamp":"2024-09-12T07:57:36.587959+0000","objects_scrubbed":4598,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 71s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802935827,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419655,"num_read_kb":1648992,"num_write":417358,"num_write_kb":1669426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"166'8852","reported_seq":14150,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517969+0000","last_change":"2024-09-12T08:59:41.685485+0000","last_active":"2024-09-12T09:00:48.517969+0000","last_peered":"2024-09-12T09:00:48.517969+0000","last_clean":"2024-09-12T09:00:48.517969+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T09:00:48.517969+0000","last_undegraded":"2024-09-12T09:00:48.517969+0000","last_fullsized":"2024-09-12T09:00:48.517969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8852","last_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_deep_scrub":"166'8852","last_deep_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_clean_scrub_stamp":"2024-09-12T08:59:41.685356+0000","objects_scrubbed":16,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:22.359185+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9038,"num_read_kb":7352,"num_write":139,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"166'54","reported_seq":431,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.672814+0000","last_change":"2024-09-12T08:59:43.643514+0000","last_active":"2024-09-12T09:00:48.672814+0000","last_peered":"2024-09-12T09:00:48.672814+0000","last_clean":"2024-09-12T09:00:48.672814+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T09:00:48.672814+0000","last_undegraded":"2024-09-12T09:00:48.672814+0000","last_fullsized":"2024-09-12T09:00:48.672814+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'54","last_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_deep_scrub":"166'54","last_deep_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_clean_scrub_stamp":"2024-09-12T08:59:43.643403+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:29:42.340048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27648966675,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701532,"num_read_kb":6687656,"num_write":1696793,"num_write_kb":6787166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13900120064,"data_stored":21950169126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40255,"ondisk_log_size":40255,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67787,"num_read_kb":56446,"num_write":1130,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44633,"internal_metadata":0},"log_size":45026,"ondisk_log_size":45026,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706432,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8040108,"kb_used_data":6788956,"kb_used_omap":1485,"kb_used_meta":1249650,"kb_avail":86331732,"statfs":{"total":96636764160,"available":88403693568,"internally_reserved":0,"allocated":6951890944,"data_stored":10975816212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1520815,"internal_metadata":1279642449},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706430,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5020188,"kb_used_data":3374604,"kb_used_omap":700,"kb_used_meta":1644867,"kb_avail":89351652,"statfs":{"total":96636764160,"available":91496091648,"internally_reserved":0,"allocated":3455594496,"data_stored":5454468819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1684344262},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100568,"kb_used_data":3417672,"kb_used_omap":704,"kb_used_meta":1682175,"kb_avail":89271272,"statfs":{"total":96636764160,"available":91413782528,"internally_reserved":0,"allocated":3499696128,"data_stored":5523263657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1722547523},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10770,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3497152512,"data_stored":5521940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452907520,"data_stored":5453144083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950060032,"data_stored":10975084563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T09:01:03.902 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:01:03.903 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:01:03.903 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-12T07:57:36.587959+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=7, tm_min=57, tm_sec=36, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=8, tm_min=59, tm_sec=38, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T09:01:03.903 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T09:01:23.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-12T09:01:24.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-12T09:01:24.210 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-12T09:01:24.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":2084,"stamp":"2024-09-12T09:01:22.695328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27649561321,"num_objects":20095,"num_object_clones":14848,"num_object_copies":40190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20095,"num_whiteouts":0,"num_read":1769375,"num_read_kb":6744148,"num_write":1698002,"num_write_kb":6788712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85401,"ondisk_log_size":85401,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18131744,"kb_used_data":13581232,"kb_used_omap":2890,"kb_used_meta":4547573,"kb_avail":264983776,"statfs":{"total":289910292480,"available":271343386624,"internally_reserved":0,"allocated":13907181568,"data_stored":21953548688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2959782,"internal_metadata":4656715354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"2.7","version":"166'6016","reported_seq":9845,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.518028+0000","last_change":"2024-09-12T08:59:44.667699+0000","last_active":"2024-09-12T09:00:48.518028+0000","last_peered":"2024-09-12T09:00:48.518028+0000","last_clean":"2024-09-12T09:00:48.518028+0000","last_became_active":"2024-09-12T07:57:29.576539+0000","last_became_peered":"2024-09-12T07:57:29.576539+0000","last_unstale":"2024-09-12T09:00:48.518028+0000","last_undegraded":"2024-09-12T09:00:48.518028+0000","last_fullsized":"2024-09-12T09:00:48.518028+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6016","last_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_deep_scrub":"166'6016","last_deep_scrub_stamp":"2024-09-12T08:59:44.667499+0000","last_clean_scrub_stamp":"2024-09-12T08:59:44.667499+0000","objects_scrubbed":21,"log_size":6016,"log_dups_size":0,"ondisk_log_size":6016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T09:41:17.115397+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0.0022618170000000002,"stat_sum":{"num_bytes":890,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6315,"num_read_kb":5116,"num_write":125,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"166'5907","reported_seq":12047,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517943+0000","last_change":"2024-09-12T08:59:45.602355+0000","last_active":"2024-09-12T09:00:48.517943+0000","last_peered":"2024-09-12T09:00:48.517943+0000","last_clean":"2024-09-12T09:00:48.517943+0000","last_became_active":"2024-09-12T07:57:29.579405+0000","last_became_peered":"2024-09-12T07:57:29.579405+0000","last_unstale":"2024-09-12T09:00:48.517943+0000","last_undegraded":"2024-09-12T09:00:48.517943+0000","last_fullsized":"2024-09-12T09:00:48.517943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5907","last_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_deep_scrub":"166'5907","last_deep_scrub_stamp":"2024-09-12T08:59:45.602237+0000","last_clean_scrub_stamp":"2024-09-12T08:59:45.602237+0000","objects_scrubbed":11,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:50:43.326131+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.001504004,"stat_sum":{"num_bytes":371,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10707,"num_read_kb":9555,"num_write":128,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3316,"num_omap_keys":70,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"166'8942","reported_seq":14463,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517929+0000","last_change":"2024-09-12T08:59:46.663069+0000","last_active":"2024-09-12T09:00:48.517929+0000","last_peered":"2024-09-12T09:00:48.517929+0000","last_clean":"2024-09-12T09:00:48.517929+0000","last_became_active":"2024-09-12T07:57:29.579341+0000","last_became_peered":"2024-09-12T07:57:29.579341+0000","last_unstale":"2024-09-12T09:00:48.517929+0000","last_undegraded":"2024-09-12T09:00:48.517929+0000","last_fullsized":"2024-09-12T09:00:48.517929+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8942","last_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_deep_scrub":"166'8942","last_deep_scrub_stamp":"2024-09-12T08:59:46.662900+0000","last_clean_scrub_stamp":"2024-09-12T08:59:46.662900+0000","objects_scrubbed":14,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:02:29.570202+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.0018139969999999999,"stat_sum":{"num_bytes":454,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8870,"num_read_kb":7237,"num_write":175,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2638,"num_omap_keys":38,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"166'8925","reported_seq":15636,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517921+0000","last_change":"2024-09-12T08:59:42.748644+0000","last_active":"2024-09-12T09:00:48.517921+0000","last_peered":"2024-09-12T09:00:48.517921+0000","last_clean":"2024-09-12T09:00:48.517921+0000","last_became_active":"2024-09-12T07:57:29.576609+0000","last_became_peered":"2024-09-12T07:57:29.576609+0000","last_unstale":"2024-09-12T09:00:48.517921+0000","last_undegraded":"2024-09-12T09:00:48.517921+0000","last_fullsized":"2024-09-12T09:00:48.517921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8925","last_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_deep_scrub":"166'8925","last_deep_scrub_stamp":"2024-09-12T08:59:42.748481+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.748481+0000","objects_scrubbed":16,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:48:03.460379+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.0011820960000000001,"stat_sum":{"num_bytes":433,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":10614,"num_read_kb":8788,"num_write":171,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3384,"num_omap_keys":79,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"167'437118","reported_seq":863585,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:01:01.257121+0000","last_change":"2024-09-12T09:01:01.257121+0000","last_active":"2024-09-12T09:01:01.257121+0000","last_peered":"2024-09-12T09:01:01.257121+0000","last_clean":"2024-09-12T09:01:01.257121+0000","last_became_active":"2024-09-12T07:57:37.611631+0000","last_became_peered":"2024-09-12T07:57:37.611631+0000","last_unstale":"2024-09-12T09:01:01.257121+0000","last_undegraded":"2024-09-12T09:01:01.257121+0000","last_fullsized":"2024-09-12T09:01:01.257121+0000","mapping_epoch":17,"log_start":"164'427042","ondisk_log_start":"164'427042","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'437118","last_scrub_stamp":"2024-09-12T09:01:01.256966+0000","last_deep_scrub":"167'437118","last_deep_scrub_stamp":"2024-09-12T09:01:01.256966+0000","last_clean_scrub_stamp":"2024-09-12T09:01:01.256966+0000","objects_scrubbed":5057,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":79,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:25:50.278772+0000","scrub_duration":78010,"objects_trimmed":0,"snaptrim_duration":0.12788956100000001,"stat_sum":{"num_bytes":6987386880,"num_objects":5057,"num_object_clones":3762,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":429547,"num_read_kb":1688668,"num_write":428514,"num_write_kb":1714056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"166'5964","reported_seq":11232,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.343361+0000","last_change":"2024-09-12T08:59:42.291476+0000","last_active":"2024-09-12T09:00:48.343361+0000","last_peered":"2024-09-12T09:00:48.343361+0000","last_clean":"2024-09-12T09:00:48.343361+0000","last_became_active":"2024-09-12T07:57:29.576344+0000","last_became_peered":"2024-09-12T07:57:29.576344+0000","last_unstale":"2024-09-12T09:00:48.343361+0000","last_undegraded":"2024-09-12T09:00:48.343361+0000","last_fullsized":"2024-09-12T09:00:48.343361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'5964","last_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_deep_scrub":"166'5964","last_deep_scrub_stamp":"2024-09-12T08:59:42.291334+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.291334+0000","objects_scrubbed":17,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:52:22.139064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0.0020453540000000001,"stat_sum":{"num_bytes":571,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7672,"num_read_kb":6537,"num_write":167,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1903,"num_omap_keys":26,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"167'434216","reported_seq":857908,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:01:04.620859+0000","last_change":"2024-09-12T09:01:04.620859+0000","last_active":"2024-09-12T09:01:04.620859+0000","last_peered":"2024-09-12T09:01:04.620859+0000","last_clean":"2024-09-12T09:01:04.620859+0000","last_became_active":"2024-09-12T07:57:37.611775+0000","last_became_peered":"2024-09-12T07:57:37.611775+0000","last_unstale":"2024-09-12T09:01:04.620859+0000","last_undegraded":"2024-09-12T09:01:04.620859+0000","last_fullsized":"2024-09-12T09:01:04.620859+0000","mapping_epoch":17,"log_start":"164'424136","ondisk_log_start":"164'424136","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'434216","last_scrub_stamp":"2024-09-12T09:01:04.620758+0000","last_deep_scrub":"167'434216","last_deep_scrub_stamp":"2024-09-12T09:01:04.620758+0000","last_clean_scrub_stamp":"2024-09-12T09:01:04.620758+0000","objects_scrubbed":5002,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":78,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:38:01.257547+0000","scrub_duration":77062,"objects_trimmed":0,"snaptrim_duration":0.12670109099999999,"stat_sum":{"num_bytes":6930407424,"num_objects":5002,"num_object_clones":3719,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":426731,"num_read_kb":1677728,"num_write":425638,"num_write_kb":1702552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"166'161","reported_seq":3721,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.409802+0000","last_change":"2024-09-12T08:59:42.976412+0000","last_active":"2024-09-12T09:00:48.409802+0000","last_peered":"2024-09-12T09:00:48.409802+0000","last_clean":"2024-09-12T09:00:48.409802+0000","last_became_active":"2024-09-12T07:57:29.576225+0000","last_became_peered":"2024-09-12T07:57:29.576225+0000","last_unstale":"2024-09-12T09:00:48.409802+0000","last_undegraded":"2024-09-12T09:00:48.409802+0000","last_fullsized":"2024-09-12T09:00:48.409802+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'161","last_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_deep_scrub":"166'161","last_deep_scrub_stamp":"2024-09-12T08:59:42.976148+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.976148+0000","objects_scrubbed":14,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:11:38.757967+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0010356689999999999,"stat_sum":{"num_bytes":573,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5969,"num_read_kb":4845,"num_write":88,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"167'433741","reported_seq":856305,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:58.110106+0000","last_change":"2024-09-12T09:00:58.110106+0000","last_active":"2024-09-12T09:00:58.110106+0000","last_peered":"2024-09-12T09:00:58.110106+0000","last_clean":"2024-09-12T09:00:58.110106+0000","last_became_active":"2024-09-12T07:57:37.606146+0000","last_became_peered":"2024-09-12T07:57:37.606146+0000","last_unstale":"2024-09-12T09:00:58.110106+0000","last_undegraded":"2024-09-12T09:00:58.110106+0000","last_fullsized":"2024-09-12T09:00:58.110106+0000","mapping_epoch":17,"log_start":"164'423642","ondisk_log_start":"164'423642","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'433741","last_scrub_stamp":"2024-09-12T09:00:58.109963+0000","last_deep_scrub":"167'433741","last_deep_scrub_stamp":"2024-09-12T09:00:58.109963+0000","last_clean_scrub_stamp":"2024-09-12T09:00:58.109963+0000","objects_scrubbed":4997,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":77,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:45:16.158082+0000","scrub_duration":76862,"objects_trimmed":0,"snaptrim_duration":0.052290822000000001,"stat_sum":{"num_bytes":6928236544,"num_objects":4997,"num_object_clones":3714,"num_object_copies":9994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":425599,"num_read_kb":1672268,"num_write":425283,"num_write_kb":1701132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"166'259","reported_seq":5502,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.410743+0000","last_change":"2024-09-12T08:59:42.002273+0000","last_active":"2024-09-12T09:00:48.410743+0000","last_peered":"2024-09-12T09:00:48.410743+0000","last_clean":"2024-09-12T09:00:48.410743+0000","last_became_active":"2024-09-12T07:57:29.578851+0000","last_became_peered":"2024-09-12T07:57:29.578851+0000","last_unstale":"2024-09-12T09:00:48.410743+0000","last_undegraded":"2024-09-12T09:00:48.410743+0000","last_fullsized":"2024-09-12T09:00:48.410743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'259","last_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_deep_scrub":"166'259","last_deep_scrub_stamp":"2024-09-12T08:59:42.002115+0000","last_clean_scrub_stamp":"2024-09-12T08:59:42.002115+0000","objects_scrubbed":14,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:48:42.106429+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00082566300000000003,"stat_sum":{"num_bytes":452,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8602,"num_read_kb":7016,"num_write":137,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2639,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"167'425792","reported_seq":842469,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:01:04.401152+0000","last_change":"2024-09-12T09:01:04.401152+0000","last_active":"2024-09-12T09:01:04.401152+0000","last_peered":"2024-09-12T09:01:04.401152+0000","last_clean":"2024-09-12T09:01:04.401152+0000","last_became_active":"2024-09-12T07:57:37.611985+0000","last_became_peered":"2024-09-12T07:57:37.611985+0000","last_unstale":"2024-09-12T09:01:04.401152+0000","last_undegraded":"2024-09-12T09:01:04.401152+0000","last_fullsized":"2024-09-12T09:01:04.401152+0000","mapping_epoch":17,"log_start":"164'415726","ondisk_log_start":"164'415726","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'425792","last_scrub_stamp":"2024-09-12T09:01:04.401005+0000","last_deep_scrub":"167'425792","last_deep_scrub_stamp":"2024-09-12T09:01:04.401005+0000","last_clean_scrub_stamp":"2024-09-12T09:01:04.401005+0000","objects_scrubbed":4914,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":76,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:58:51.770987+0000","scrub_duration":75798,"objects_trimmed":0,"snaptrim_duration":0.019955062999999999,"stat_sum":{"num_bytes":6802935827,"num_objects":4914,"num_object_clones":3653,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":419655,"num_read_kb":1648992,"num_write":417358,"num_write_kb":1669426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"166'8852","reported_seq":14150,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.517969+0000","last_change":"2024-09-12T08:59:41.685485+0000","last_active":"2024-09-12T09:00:48.517969+0000","last_peered":"2024-09-12T09:00:48.517969+0000","last_clean":"2024-09-12T09:00:48.517969+0000","last_became_active":"2024-09-12T07:57:29.576061+0000","last_became_peered":"2024-09-12T07:57:29.576061+0000","last_unstale":"2024-09-12T09:00:48.517969+0000","last_undegraded":"2024-09-12T09:00:48.517969+0000","last_fullsized":"2024-09-12T09:00:48.517969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8852","last_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_deep_scrub":"166'8852","last_deep_scrub_stamp":"2024-09-12T08:59:41.685356+0000","last_clean_scrub_stamp":"2024-09-12T08:59:41.685356+0000","objects_scrubbed":16,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:59:22.359185+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0.0013046240000000001,"stat_sum":{"num_bytes":534,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9038,"num_read_kb":7352,"num_write":139,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"166'54","reported_seq":431,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-12T09:00:48.672814+0000","last_change":"2024-09-12T08:59:43.643514+0000","last_active":"2024-09-12T09:00:48.672814+0000","last_peered":"2024-09-12T09:00:48.672814+0000","last_clean":"2024-09-12T09:00:48.672814+0000","last_became_active":"2024-09-12T07:57:26.393452+0000","last_became_peered":"2024-09-12T07:57:26.393452+0000","last_unstale":"2024-09-12T09:00:48.672814+0000","last_undegraded":"2024-09-12T09:00:48.672814+0000","last_fullsized":"2024-09-12T09:00:48.672814+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'54","last_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_deep_scrub":"166'54","last_deep_scrub_stamp":"2024-09-12T08:59:43.643403+0000","last_clean_scrub_stamp":"2024-09-12T08:59:43.643403+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:29:42.340048+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27648966675,"num_objects":19970,"num_object_clones":14848,"num_object_copies":39940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19970,"num_whiteouts":0,"num_read":1701532,"num_read_kb":6687656,"num_write":1696793,"num_write_kb":6787166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13900120064,"data_stored":21950169126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40321,"ondisk_log_size":40321,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4278,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":67787,"num_read_kb":56446,"num_write":1130,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20544,"num_omap_keys":342,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":8556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44633,"internal_metadata":0},"log_size":45026,"ondisk_log_size":45026,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706436,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8027628,"kb_used_data":6788956,"kb_used_omap":1485,"kb_used_meta":1237170,"kb_avail":86344212,"statfs":{"total":96636764160,"available":88416473088,"internally_reserved":0,"allocated":6951890944,"data_stored":10975816212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1520815,"internal_metadata":1266862929},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706434,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5003548,"kb_used_data":3374604,"kb_used_omap":700,"kb_used_meta":1628227,"kb_avail":89368292,"statfs":{"total":96636764160,"available":91513131008,"internally_reserved":0,"allocated":3455594496,"data_stored":5454468819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":717370,"internal_metadata":1667304902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100568,"kb_used_data":3417672,"kb_used_omap":704,"kb_used_meta":1682175,"kb_avail":89271272,"statfs":{"total":96636764160,"available":91413782528,"internally_reserved":0,"allocated":3499696128,"data_stored":5523263657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721597,"internal_metadata":1722547523},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":2719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14741,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":4278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19122,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10770,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3497152512,"data_stored":5521940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3452907520,"data_stored":5453144083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6950060032,"data_stored":10975084563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T09:01:24.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-12T09:01:24.535 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-12T09:01:24.535 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-12T09:01:24.536 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-12T09:01:24.536 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T09:01:27.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-12T09:01:30.637 INFO:tasks.ceph.osd.0:Stopped 2024-09-12T09:01:30.638 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-12T09:01:30.638 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T09:01:33.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-12T09:01:33.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-12T09:01:36.739 INFO:tasks.ceph.osd.1:Stopped 2024-09-12T09:01:36.740 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-12T09:01:36.740 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T09:01:38.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-12T09:01:38.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-12T09:01:42.841 INFO:tasks.ceph.osd.2:Stopped 2024-09-12T09:01:42.842 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-12T09:01:42.842 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-12T09:01:42.842 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T09:01:42.875 INFO:tasks.ceph.mgr.x:Stopped 2024-09-12T09:01:42.875 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-12T09:01:42.875 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-12T09:01:42.875 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T09:01:42.941 INFO:tasks.ceph.mon.a:Stopped 2024-09-12T09:01:42.942 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-12T09:01:42.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T09:01:43.050 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T09:01:43.050 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-12T09:01:43.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-12T09:01:43.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T09:01:43.474 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-12T09:01:43.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T09:01:43.887 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-12T09:01:44.406 INFO:tasks.ceph:Archiving mon data... 2024-09-12T09:01:44.407 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438/data/mon.a.tgz 2024-09-12T09:01:44.408 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-12T09:01:44.427 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.lP6QfrGYIf 2024-09-12T09:01:44.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.lP6QfrGYIf 2024-09-12T09:01:44.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.lP6QfrGYIf 2024-09-12T09:01:44.712 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.lP6QfrGYIf is 893KB 2024-09-12T09:01:44.784 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.lP6QfrGYIf 2024-09-12T09:01:44.789 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-12T09:01:44.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T09:01:44.932 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-12T09:01:44.932 INFO:tasks.ceph:Archiving crash dumps... 2024-09-12T09:01:44.934 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438/remote/smithi137/crash 2024-09-12T09:01:44.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-12T09:01:44.986 INFO:tasks.ceph:Compressing logs... 2024-09-12T09:01:44.986 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T09:01:45.053 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40803.log 2024-09-12T09:01:45.053 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33201.log 2024-09-12T09:01:45.053 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40803.log.gz 2024-09-12T09:01:45.053 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43357.log 2024-09-12T09:01:45.055 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33201.log.gz 2024-09-12T09:01:45.055 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45876.log 2024-09-12T09:01:45.055 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37678.log 2024-09-12T09:01:45.055 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43357.log.gz 2024-09-12T09:01:45.055 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42587.log 2024-09-12T09:01:45.055 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45876.log.gz 2024-09-12T09:01:45.056 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44979.log 2024-09-12T09:01:45.056 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37678.log: /var/log/ceph/ceph-client.admin.42587.log: gzip 0.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.37678.log.gz /var/log/ceph/ceph-client.admin.36715.log 2024-09-12T09:01:45.056 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:01:45.056 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42587.log.gz 2024-09-12T09:01:45.056 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44979.log.gz 2024-09-12T09:01:45.056 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44371.log 2024-09-12T09:01:45.077 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35202.log 2024-09-12T09:01:45.077 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36715.log.gz 2024-09-12T09:01:45.077 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-09-12T09:01:45.077 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44371.log.gz 2024-09-12T09:01:45.077 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44996.log 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40499.log 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33841.log 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44996.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40499.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32365.log 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42990.log 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33841.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32365.log.gz 2024-09-12T09:01:45.078 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38097.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42990.log.gz 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36174.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34242.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38097.log.gz 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36174.log.gz 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32120.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34802.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32120.log.gz 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42740.log 2024-09-12T09:01:45.079 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34802.log.gz 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42740.log.gz 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44198.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45200.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44198.log.gz 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40986.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45200.log.gz 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44339.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32218.log 2024-09-12T09:01:45.080 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40986.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44339.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38200.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37386.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32218.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38200.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31767.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34619.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37386.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31767.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30281.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34619.log.gz 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30247.log 2024-09-12T09:01:45.081 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30281.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44467.log 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30247.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44467.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33320.log 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33320.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38687.log 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32069.log 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-09-12T09:01:45.082 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38687.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34832.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43847.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32069.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34832.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38467.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29398.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43847.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38467.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37757.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29398.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30983.log 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37757.log.gz 2024-09-12T09:01:45.083 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32305.log 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29320.log 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30983.log.gz 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32305.log.gz 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45030.log 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29320.log.gz 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45030.log.gz 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30227.log 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30966.log 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.30227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30227.log.gz 2024-09-12T09:01:45.084 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34027.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40068.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30966.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31037.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34027.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31037.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38524.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32194.log 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38524.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-09-12T09:01:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32680.log 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30713.log 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32194.log.gz 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36742.log 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32680.log.gz 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30713.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.30713.log.gz 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr: -- /var/log/ceph/ceph-client.admin.33750.log 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37978.log 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36742.log.gz 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32731.log 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33750.log.gz 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37978.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37978.log.gz --verbose 2024-09-12T09:01:45.086 INFO:teuthology.orchestra.run.smithi137.stderr: -- /var/log/ceph/ceph-client.admin.37730.log 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30264.log 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32731.log.gz 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30659.log 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37730.log.gz 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30264.log.gz 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-09-12T09:01:45.243 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30659.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44443.log 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45400.log 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36497.log 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44443.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45400.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43265.log 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36497.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38655.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37266.log 2024-09-12T09:01:45.244 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43265.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38554.log 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37266.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39699.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45251.log 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38554.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45251.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30588.log 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42282.log 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30588.log.gz 2024-09-12T09:01:45.245 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36100.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30082.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42282.log.gz 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36100.log.gz 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32041.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30082.log.gz 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32041.log.gz 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30787.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31851.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gz 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30787.log.gz 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-09-12T09:01:45.246 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31851.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45568.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40935.log: /var/log/ceph/ceph-client.admin.45568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45568.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31557.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42417.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31557.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42832.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42417.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42832.log.gz 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31057.log 2024-09-12T09:01:45.247 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33567.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36805.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31057.log.gz 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32335.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33567.log.gz 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35105.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36805.log.gz 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32792.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32335.log.gz 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42373.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35105.log.gz 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37818.log 2024-09-12T09:01:45.248 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32792.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45132.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42373.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44829.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37818.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35449.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45132.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33290.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44829.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45285.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35449.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25248.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33290.log.gz 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31893.log 2024-09-12T09:01:45.249 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45285.log.gz 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.25248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43633.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34589.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31893.log.gz 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37022.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43633.log.gz 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40008.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34589.log.gz 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30318.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37022.log.gz 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40008.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40008.log.gz 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30136.log 2024-09-12T09:01:45.250 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30318.log.gz 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32409.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38986.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30136.log.gz 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.32409.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29240.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32409.log.gz 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38127.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38986.log.gz 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34215.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29240.log.gz 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45931.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38127.log.gz 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39014.log 2024-09-12T09:01:45.251 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34215.log.gz 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45931.log.gz 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39014.log.gz 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33641.log 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39479.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37236.log 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2024-09-12T09:01:45.252 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33641.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.37236.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29182.log 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 29.5% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37236.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38890.log 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43787.log 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29182.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29110.log 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38890.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43787.log.gz 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45268.log 2024-09-12T09:01:45.253 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29110.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35931.log 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45268.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29156.log 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37878.log 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35931.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44659.log 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29156.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37878.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37878.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44659.log.gz 2024-09-12T09:01:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33177.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37634.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43459.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33177.log.gz 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42006.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37634.log.gz 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39046.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43459.log.gz 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39323.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42006.log.gz 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39046.log.gz 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33506.log 2024-09-12T09:01:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39323.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43879.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33044.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33506.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35396.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43879.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38172.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33044.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35396.log: /var/log/ceph/ceph-client.admin.40907.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35396.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38172.log.gz 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30355.log 2024-09-12T09:01:45.256 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44858.log 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.30355.log: /var/log/ceph/ceph-client.admin.39918.log 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30355.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44858.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30116.log 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45388.log 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30116.log.gz 2024-09-12T09:01:45.257 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45388.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45388.log.gz /var/log/ceph/ceph-client.admin.35602.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38711.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35045.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35602.log.gz 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38711.log.gz 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35045.log: -- /var/log/ceph/ceph-client.admin.40175.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35045.log.gz 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43155.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31431.log 2024-09-12T09:01:45.258 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40175.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43155.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36070.log 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36070.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45496.log 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-09-12T09:01:45.259 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33449.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30858.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45496.log.gz 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30912.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33449.log.gz 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44035.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30858.log.gz 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44926.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30912.log.gz 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30372.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44035.log.gz 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31809.log 2024-09-12T09:01:45.260 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44926.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45064.log 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30372.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42096.log 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31809.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45064.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44309.log 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42096.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33014.log 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37509.log 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44309.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25248.log.gz 2024-09-12T09:01:45.261 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33014.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45436.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37509.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30409.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45436.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30696.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30409.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34119.log 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2024-09-12T09:01:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33810.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30696.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34305.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34119.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33810.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34471.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34305.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32518.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34471.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32488.log 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2024-09-12T09:01:45.263 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35793.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32518.log.gz 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45183.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32488.log.gz 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35793.log.gz 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45183.log.gz 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29085.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5/var/log/ceph/ceph-client.admin.35352.log: --verbose -- /var/log/ceph/ceph-client.admin.35763.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29085.log.gz 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36040.log 2024-09-12T09:01:45.264 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35763.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5/var/log/ceph/ceph-client.admin.44711.log: --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37458.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36040.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45745.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35632.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37458.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36343.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45745.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36283.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35632.log.gz 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41081.log 2024-09-12T09:01:45.265 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36343.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30733.log 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36283.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41949.log 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41081.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32904.log 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30733.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31473.log 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41949.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34663.log 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32904.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.31473.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34958.log 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31473.log.gz 2024-09-12T09:01:45.266 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44065.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34663.log.gz 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39383.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34958.log.gz 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44065.log.gz 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30193.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39383.log.gz 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37926.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38934.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30193.log.gz 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29316.log 2024-09-12T09:01:45.267 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37926.log.gz 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41925.log 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38934.log.gz 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.29316.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33074.log 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29316.log.gz 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42224.log 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41925.log.gz 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36313.log 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33074.log.gz 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31003.log 2024-09-12T09:01:45.268 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42224.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42661.log 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36313.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32957.log 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31003.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42661.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32957.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34163.log 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39235.log: /var/log/ceph/ceph-client.admin.40663.log 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34163.log.gz 2024-09-12T09:01:45.269 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43234.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35959.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40663.log.gz 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35230.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43234.log.gz 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35959.log.gz 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38371.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38415.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35230.log.gz 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38371.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38371.log.gz 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43021.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38415.log.gz 2024-09-12T09:01:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32578.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43021.log.gz 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30605.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36470.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32578.log.gz 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33149.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30605.log.gz 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36470.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.36470.log.gz /var/log/ceph/ceph-client.admin.44170.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29468.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33149.log.gz 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30463.log 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-09-12T09:01:45.271 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34906.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29468.log.gz 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42469.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30463.log.gz 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39753.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34906.log.gz 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32548.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42469.log.gz 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44279.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39753.log.gz 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30750.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32548.log.gz 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40203.log 2024-09-12T09:01:45.272 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44279.log.gz 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30679.log 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30750.log.gz 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36253.log/var/log/ceph/ceph-client.admin.40203.log: 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40203.log.gz 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30679.log.gz 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36253.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.36253.log.gz /var/log/ceph/ceph-client.admin.42172.log 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32704.log 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35983.log 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42172.log.gz 2024-09-12T09:01:45.273 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32704.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32704.log.gz 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35983.log.gz 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41789.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.41484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30949.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45849.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29397.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30949.log.gz 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45849.log.gz 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29397.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.29397.log.gz -- 2024-09-12T09:01:45.274 INFO:teuthology.orchestra.run.smithi137.stderr: /var/log/ceph/ceph-client.admin.45424.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37848.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45424.log.gz 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32166.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37848.log.gz 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30767.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43326.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32166.log.gz 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45234.log 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30767.log.gz 2024-09-12T09:01:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38443.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43326.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43081.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45234.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38005.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38443.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34059.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43081.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38005.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29494.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34059.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32852.log 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2024-09-12T09:01:45.276 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37185.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29494.log.gz 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30156.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32852.log.gz 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36772.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37185.log.gz 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43572.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30156.log.gz 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45115.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36772.log.gz 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43572.log.gz 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45115.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.45115.log.gz --verbose 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr: -- /var/log/ceph/ceph-client.admin.25224.log 2024-09-12T09:01:45.277 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30443.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.25224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25224.log.gz 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45589.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30443.log.gz 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41514.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44249.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45589.log.gz 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33994.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43730.log 2024-09-12T09:01:45.278 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33994.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43730.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.31599.log -- replaced with /var/log/ceph/ceph-client.admin.43730.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30517.log 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31599.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43296.log 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30517.log.gz 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41155.log 2024-09-12T09:01:45.279 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43387.log 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43296.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41155.log: /var/log/ceph/ceph-client.admin.43705.log 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41155.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43387.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32822.log 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45905.log 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43705.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36618.log 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32822.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45905.log.gz 2024-09-12T09:01:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36618.log.gz 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44738.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33597.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44738.log.gz 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33597.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33597.log.gz 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30821.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31987.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37570.log 2024-09-12T09:01:45.281 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30821.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43542.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31987.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37570.log: -- /var/log/ceph/ceph-client.admin.34715.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37570.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37209.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43542.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33397.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34715.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42312.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37209.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45098.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33397.log.gz 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33476.log 2024-09-12T09:01:45.282 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42312.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42771.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45098.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39980.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33476.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42771.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39980.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39980.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42801.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34985.log 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2024-09-12T09:01:45.283 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42801.log.gz 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30065.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34985.log.gz 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30065.log.gz 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31263.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29545.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36373.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31263.log.gz 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39107.log 2024-09-12T09:01:45.284 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29545.log.gz 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36373.log.gzgzip 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33104.log 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37788.log 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39107.log.gz 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33260.log 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33104.log.gz 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37788.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37788.log.gz -5 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38585.log 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36965.log 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33260.log.gz 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38585.log.gz 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36965.log.gz 2024-09-12T09:01:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45556.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40619.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36691.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45556.log.gz 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40619.log: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40619.log.gz -5 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29520.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38037.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44415.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36691.log.gz 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29520.log.gz 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39506.log 2024-09-12T09:01:45.286 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38037.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46007.log 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44415.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35260.log 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39506.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46007.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21536.log 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46007.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45544.log 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35260.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.tmp-client.admin.21536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21536.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31128.log 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45544.log.gz 2024-09-12T09:01:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45983.log 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31128.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31074.log 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45983.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45149.log 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31231.log 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31074.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45149.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37540.log 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31231.log.gz 2024-09-12T09:01:45.288 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31145.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30895.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37540.log.gz 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42689.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31145.log.gz 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30895.log.gz 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42689.log.gz 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31725.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-09-12T09:01:45.289 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31725.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45460.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43051.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36663.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45460.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43051.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32437.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36663.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45823.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34691.log 2024-09-12T09:01:45.290 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32437.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33915.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45823.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34691.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43484.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33915.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27894.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43484.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45364.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.27894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27894.log.gz 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44768.log 2024-09-12T09:01:45.291 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45364.log.gz 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33943.log 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30841.log 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44768.log.gz 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43602.log 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33943.log.gz 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31295.log 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30841.log.gz 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43602.log.gz 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39455.log 2024-09-12T09:01:45.292 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31295.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30301.log 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45484.log 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39455.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40098.log/var/log/ceph/ceph-client.admin.30301.log: 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30301.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36992.log 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45484.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40098.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35075.log 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5/var/log/ceph/ceph-client.admin.36992.log: --verbose -- /var/log/ceph/ceph-client.admin.36226.log 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36992.log.gz 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2024-09-12T09:01:45.293 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35075.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36226.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.42066.log -- replaced with /var/log/ceph/ceph-client.admin.36226.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45166.log 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37113.log 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45166.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34365.log 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37113.log: /var/log/ceph/ceph-client.admin.32933.log 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37113.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31111.log 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34365.log.gz 2024-09-12T09:01:45.294 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45520.log 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32933.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41234.log/var/log/ceph/ceph-client.admin.31111.log: 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31111.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45520.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30210.log 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30210.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30210.log.gz 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34335.log 2024-09-12T09:01:45.295 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32608.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34335.log.gz 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31091.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36130.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32608.log.gz 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38615.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31091.log.gz 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29208.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36130.log.gz 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32984.log 2024-09-12T09:01:45.296 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38615.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29208.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45719.log 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32984.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42342.log 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45719.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25200.log 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42342.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45957.log 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.25200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43677.log 0.0% 2024-09-12T09:01:45.297 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25200.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45957.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45957.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43677.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39946.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43975.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33353.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39946.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43975.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35541.log 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33353.log.gz 2024-09-12T09:01:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30497.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38829.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35541.log.gz 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45797.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30497.log.gz 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30173.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38829.log.gz 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45797.log.gz 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43951.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30173.log.gz 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42200.log 2024-09-12T09:01:45.299 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31641.log 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43951.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33693.log 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42200.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36528.log 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31641.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36911.log 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33693.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42862.log 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36528.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30804.log 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36911.log.gz 2024-09-12T09:01:45.300 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25347.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42862.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.30804.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30804.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.25347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35684.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25347.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42557.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34273.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35684.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39536.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42557.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5/var/log/ceph/ceph-client.admin.34273.log: --verbose -- /var/log/ceph/ceph-client.admin.39671.log 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34273.log.gz 2024-09-12T09:01:45.301 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29390.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39536.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39671.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29390.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40743.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38769.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40743.log.gz 2024-09-12T09:01:45.302 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35510.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38769.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37053.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36588.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35510.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37053.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42906.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31388.log 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42906.log.gz 2024-09-12T09:01:45.303 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31020.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45013.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31388.log.gz 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37083.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31020.log.gz 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35290.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45013.log.gz 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33669.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37083.log.gz 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35290.log.gz 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30875.log 2024-09-12T09:01:45.304 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33669.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34772.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33780.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30875.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32652.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34772.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34742.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33780.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45303.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32652.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25105.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34742.log.gz 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45320.log 2024-09-12T09:01:45.305 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45303.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.25105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37296.log 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25105.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37482.log 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45320.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37296.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30571.log 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37482.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39353.log 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5/var/log/ceph/ceph-client.admin.30571.log: --verbose -- /var/log/ceph/ceph-client.admin.29242.log 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30571.log.gz 2024-09-12T09:01:45.306 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44494.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39353.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42713.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29242.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33967.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44494.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35712.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42713.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33967.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41676.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35712.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41820.log 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-09-12T09:01:45.307 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41676.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41820.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40589.log: -- /var/log/ceph/ceph-client.admin.34528.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37356.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32093.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34528.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30099.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37356.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30335.log 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32093.log.gz 2024-09-12T09:01:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30099.log.gz 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35823.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30335.log.gz 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39723.log: /var/log/ceph/ceph-client.admin.35823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35887.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35823.log.gz 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45667.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36866.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35887.log.gz 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35178.log 2024-09-12T09:01:45.309 INFO:teuthology.orchestra.run.smithi137.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.45667.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34446.log 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36866.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35178.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36417.log 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34446.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29315.log 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45693.log 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36417.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36836.log 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29315.log.gz 2024-09-12T09:01:45.310 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30551.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45693.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31515.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36836.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36445.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30551.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36010.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31515.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44876.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36445.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36010.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30480.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45641.log 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44876.log.gz 2024-09-12T09:01:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29571.log 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30480.log.gz 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33229.log 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45641.log.gz 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25283.log 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29571.log.gz 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33229.log.gz 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.25283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2024-09-12T09:01:45.312 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25283.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29241.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33883.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30642.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29241.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38738.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33883.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30642.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31183.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38738.log.gz 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44096.log 2024-09-12T09:01:45.313 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31183.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30389.log 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44096.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40448.log 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30426.log 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30389.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40038.log 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40448.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30426.log.gz 2024-09-12T09:01:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39843.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40038.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42934.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31165.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39843.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36939.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42934.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31345.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31165.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41111.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36939.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.31345.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45352.log 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31345.log.gz 2024-09-12T09:01:45.315 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35854.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41111.log.gz 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37604.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45352.log.gz 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33720.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35854.log.gz 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32245.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37604.log.gz 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44687.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33720.log.gz 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37153.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32245.log.gz 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38311.log 2024-09-12T09:01:45.316 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44687.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34934.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37153.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35736.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38311.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34934.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29130.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35736.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32461.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45081.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29130.log.gz 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31935.log 2024-09-12T09:01:45.317 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32461.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33537.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45081.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43183.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31935.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32761.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33537.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30625.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43183.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32761.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31683.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30625.log.gz 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45376.log 2024-09-12T09:01:45.318 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44222.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31683.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45771.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45376.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43207.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44222.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34498.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45771.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43207.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43111.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34498.log.gz 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2024-09-12T09:01:45.319 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32275.log 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43111.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37326.log 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32275.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35424.log 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30534.log 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37326.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29060.log 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35424.log.gz 2024-09-12T09:01:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.30534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34089.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30534.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36558.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29060.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34089.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36558.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35320.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44961.log 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35320.log.gz 2024-09-12T09:01:45.321 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43757.log 2024-09-12T09:01:45.322 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2024-09-12T09:01:45.322 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44961.log.gz 2024-09-12T09:01:45.322 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43757.log.gz 2024-09-12T09:01:45.322 INFO:teuthology.orchestra.run.smithi137.stderr: 90.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-12T09:01:54.758 INFO:teuthology.orchestra.run.smithi137.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-12T09:01:55.941 INFO:teuthology.orchestra.run.smithi137.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-12T09:27:02.726 INFO:teuthology.orchestra.run.smithi137.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-12T09:34:23.553 INFO:teuthology.orchestra.run.smithi137.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-12T09:34:27.155 INFO:teuthology.orchestra.run.smithi137.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-12T09:34:27.156 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-12T09:34:27.156 INFO:teuthology.orchestra.run.smithi137.stderr:real 32m42.126s 2024-09-12T09:34:27.157 INFO:teuthology.orchestra.run.smithi137.stderr:user 31m38.242s 2024-09-12T09:34:27.157 INFO:teuthology.orchestra.run.smithi137.stderr:sys 1m39.981s 2024-09-12T09:34:27.158 INFO:tasks.ceph:Archiving logs... 2024-09-12T09:34:27.159 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438/remote/smithi137/log 2024-09-12T09:34:27.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T09:35:54.834 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-12T09:35:54.845 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-12T09:35:54.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T09:35:54.892 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-12T09:35:54.988 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-12T09:35:54.988 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T09:35:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2024-09-12T09:35:55.060 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2024-09-12T09:35:55.060 INFO:teuthology.orchestra.run.smithi137.stdout:+hv01.front.sepi 67.205.162.81 3 u 85 256 377 0.196 +0.587 0.398 2024-09-12T09:35:55.060 INFO:teuthology.orchestra.run.smithi137.stdout:*hv02.front.sepi 63.231.80.2 3 u 54 256 377 0.205 -3.209 0.170 2024-09-12T09:35:55.060 INFO:teuthology.orchestra.run.smithi137.stdout:+hv03.front.sepi 74.6.168.72 3 u 68 256 377 0.217 -4.746 0.220 2024-09-12T09:35:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-12T09:35:55.061 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-12T09:35:55.071 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-12T09:35:55.072 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-12T09:35:55.103 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-12T09:35:55.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-12T09:35:55.119 INFO:teuthology.task.internal:Duration was 6495.608849 seconds 2024-09-12T09:35:55.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-12T09:35:55.128 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-12T09:35:55.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T09:35:55.392 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-12T09:35:55.392 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-09-12T09:35:55.392 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T09:35:55.449 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-12T09:35:55.449 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T09:35:55.541 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-12T09:35:55.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T09:35:55.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-12T09:35:55.875 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-12T09:35:55.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T09:35:55.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-12T09:35:55.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T09:35:55.974 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-09-12T09:35:56.013 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T09:35:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T09:35:56.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-12T09:35:56.070 INFO:teuthology.task.internal:Transferring archived files... 2024-09-12T09:35:56.072 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902438/remote/smithi137 2024-09-12T09:35:56.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T09:35:56.126 INFO:teuthology.task.internal:Removing archive directory... 2024-09-12T09:35:56.126 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T09:35:56.173 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-12T09:35:56.184 INFO:teuthology.task.internal:Not uploading archives. 2024-09-12T09:35:56.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-12T09:35:56.194 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-12T09:35:56.194 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T09:35:56.220 INFO:teuthology.orchestra.run.smithi137.stdout: 658466 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 09:35 /home/ubuntu/cephtest 2024-09-12T09:35:56.222 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-12T09:35:56.232 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-12T09:35:56.254 INFO:teuthology.run:Summary data: description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 6495.608848571777 flavor: default owner: scheduled_dis@teuthology success: true 2024-09-12T09:35:56.255 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T09:35:56.381 INFO:teuthology.run:pass