2024-09-13T07:14:55.438 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T07:14:55.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T07:14:55.542 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930 branch: reef description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_20.04} workloads/rbd_cli_generic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900930' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: client: rbd default data pool: datapool rbd default features: 125 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 7a853b1706d076d58e0da7a5d3b9b9eb36276880 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 248 sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 sleep_before_teardown: 0 subset: 116/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7a853b1706d076d58e0da7a5d3b9b9eb36276880 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - workunit: clients: client.0: - rbd/cli_generic.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_22:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T07:14:55.542 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa; will attempt to use it 2024-09-13T07:14:55.543 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa/tasks 2024-09-13T07:14:55.543 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T07:14:55.544 INFO:teuthology.task.internal:Checking packages... 2024-09-13T07:14:55.567 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '48a7de7e6e1701a88fd571c7e34908ac4b8ed412' 2024-09-13T07:14:55.567 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T07:14:55.567 INFO:teuthology.packaging:ref: None 2024-09-13T07:14:55.567 INFO:teuthology.packaging:tag: None 2024-09-13T07:14:55.568 INFO:teuthology.packaging:branch: reef 2024-09-13T07:14:55.568 INFO:teuthology.packaging:sha1: 48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T07:14:55.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-13T07:14:55.836 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-843-g2c01458a-1focal 2024-09-13T07:14:55.838 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T07:14:55.845 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T07:14:55.845 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T07:14:55.855 INFO:teuthology.task.internal:Saving configuration 2024-09-13T07:14:55.866 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T07:14:55.873 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T07:14:55.908 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 07:09:09.117943', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-13T07:14:55.908 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T07:14:55.915 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-13T07:14:55.915 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T07:14:55.960 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6358f0e50>, signals=[15]) 2024-09-13T07:14:55.960 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T07:14:55.966 INFO:teuthology.task.internal:Opening connections... 2024-09-13T07:14:55.966 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:14:55.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T07:14:56.076 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T07:14:56.083 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-09-13T07:14:56.199 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-09-13T07:14:56.199 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-09-13T07:14:56.205 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2024-09-13T07:14:56.205 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-13T07:14:56.205 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="20.04" 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=focal 2024-09-13T07:14:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=focal 2024-09-13T07:14:56.207 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-09-13T07:14:56.244 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T07:14:56.253 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T07:14:56.258 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T07:14:56.258 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-09-13T07:14:56.264 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T07:14:56.270 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T07:14:56.271 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-09-13T07:14:56.320 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T07:14:56.357 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T07:14:56.379 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T07:14:56.379 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T07:14:56.380 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T07:14:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T07:14:56.387 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-13T07:14:56.394 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-124-generic 2024-09-13T07:14:56.394 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.4.0-124-generic 2024-09-13T07:14:56.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-09-13T07:14:56.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-13T07:14:56.683 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-13T07:14:56.801 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T07:14:56.883 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-13T07:14:56.987 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-13T07:14:57.073 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-13T07:14:57.117 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-13T07:14:57.120 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-13T07:14:57.121 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-13T07:14:57.141 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-13T07:14:57.144 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-13T07:14:57.147 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-13T07:14:57.147 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-13T07:14:57.154 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-13T07:14:57.158 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-13T07:14:57.160 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-13T07:14:57.160 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-13T07:14:57.160 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-13T07:14:57.169 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-13T07:14:57.169 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-13T07:14:57.236 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-13T07:14:57.377 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-13T07:14:57.621 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-13T07:14:57.822 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-13T07:14:57.836 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-13T07:14:57.837 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-13T07:14:57.933 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-13T07:14:57.933 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-13T07:14:57.948 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-13T07:14:57.949 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-13T07:14:57.968 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-13T07:14:58.007 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-13T07:14:58.013 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-13T07:14:58.014 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-13T07:14:58.014 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-13T07:14:58.066 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-13T07:14:58.076 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-13T07:14:58.084 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-13T07:14:58.086 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-13T07:14:58.090 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-13T07:14:58.150 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-13T07:14:58.150 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-13T07:14:58.151 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-13T07:15:01.671 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 21.8 MB in 3s (7,093 kB/s) 2024-09-13T07:15:03.265 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T07:15:03.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T07:15:03.382 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T07:15:03.605 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-13T07:15:03.605 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-13T07:15:03.819 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-09-13T07:15:03.819 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-13T07:15:03.819 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-13T07:15:03.820 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-13T07:15:03.821 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-09-13T07:15:03.821 INFO:teuthology.orchestra.run.smithi005.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-13T07:15:03.821 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-09-13T07:15:03.821 INFO:teuthology.orchestra.run.smithi005.stdout: thermald 2024-09-13T07:15:03.882 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-09-13T07:15:03.882 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-13T07:15:03.882 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-13T07:15:03.882 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-13T07:15:03.884 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2024-09-13T07:15:03.884 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T07:15:03.939 INFO:teuthology.orchestra.run.smithi005.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-13T07:15:03.939 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 77.2 MB of archives. 2024-09-13T07:15:03.939 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-13T07:15:03.939 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-13T07:15:04.273 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-13T07:15:04.365 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-13T07:15:04.883 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-13T07:15:04.883 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-13T07:15:04.883 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-13T07:15:04.979 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-13T07:15:04.991 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-13T07:15:05.668 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 77.2 MB in 1s (70.2 MB/s) 2024-09-13T07:15:05.842 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-13T07:15:07.573 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-13T07:15:07.578 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T07:15:07.656 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:09.506 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-13T07:15:09.520 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T07:15:09.614 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:10.718 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-13T07:15:10.732 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T07:15:10.776 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:15.525 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T07:15:15.665 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T07:15:16.008 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T07:15:16.135 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T07:15:16.448 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-13T07:15:16.463 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-13T07:15:16.529 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-13T07:15:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-13T07:15:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T07:15:22.177 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:24.996 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T07:15:25.114 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T07:15:25.450 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:26.720 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:27.915 INFO:teuthology.orchestra.run.smithi005.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-13T07:15:27.915 INFO:teuthology.orchestra.run.smithi005.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-13T07:15:28.055 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:30.282 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-13T07:15:30.408 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-13T07:15:30.534 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:30.660 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-13T07:15:30.786 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-13T07:15:30.921 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T07:15:30.987 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T07:15:30.988 INFO:teuthology.orchestra.run.smithi005.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-13T07:15:53.228 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T07:15:53.229 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub' 2024-09-13T07:15:53.246 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T07:15:53.272 INFO:teuthology.orchestra.run.smithi005.stdout:Generating grub configuration file ... 2024-09-13T07:15:53.581 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-13T07:15:53.605 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-13T07:15:53.967 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-13T07:15:53.979 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-13T07:15:55.941 INFO:teuthology.orchestra.run.smithi005.stdout:done 2024-09-13T07:15:56.365 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-09-13T07:15:56.409 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 20 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.4.0.195.193 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-13T07:15:56.410 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T07:15:56.411 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-09-13T07:15:56.411 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-09-13T07:15:56.411 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-09-13T07:15:56.411 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-09-13T07:15:56.411 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-13T07:15:56.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T07:15:56.411 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T07:15:56.411 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T07:15:56.412 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T07:15:56.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-09-13T07:15:56.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-13T07:15:56.627 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T07:15:56.627 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-13T07:15:56.634 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-13T07:15:56.652 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-13T07:15:58.551 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T07:15:58.579 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T07:15:58.650 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T07:15:58.874 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-13T07:15:58.875 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-13T07:15:59.130 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-13T07:15:59.130 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-13T07:15:59.132 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 20 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-09-13T07:15:59.153 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.4.0.195.193 2024-09-13T07:15:59.154 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-13T07:15:59.154 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T07:15:59.154 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-09-13T07:15:59.154 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-09-13T07:15:59.154 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-09-13T07:15:59.154 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-09-13T07:15:59.154 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T07:15:59.159 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.uqCcnqJjwW 2024-09-13T07:15:59.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub/grub.cfg /tmp/tmp.uqCcnqJjwW 2024-09-13T07:15:59.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.uqCcnqJjwW 2024-09-13T07:15:59.254 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.uqCcnqJjwW is 9KB 2024-09-13T07:15:59.269 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.uqCcnqJjwW 2024-09-13T07:15:59.275 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T07:15:59.331 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:15:59.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T07:15:59.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T07:15:59.361 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-13T07:15:59.361 DEBUG:teuthology.orchestra.run.smithi005:> sudo update-grub 2024-09-13T07:16:00.660 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub' 2024-09-13T07:16:00.670 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T07:16:00.679 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-09-13T07:16:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-13T07:16:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-13T07:16:01.281 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-13T07:16:01.293 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-13T07:16:02.256 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-09-13T07:16:02.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-09-13T07:16:32.289 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T07:16:32.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T07:16:32.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T07:17:32.358 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T07:17:41.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T07:17:41.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T07:17:42.071 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-13T07:17:43.127 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T07:17:43.127 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-13T07:17:43.128 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-13T07:17:43.134 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-195-generic 2024-09-13T07:17:43.134 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-13T07:17:43.134 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T07:17:43.134 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T07:17:44.135 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T07:17:44.135 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T07:17:44.263 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-09-13T07:17:44.292 DEBUG:teuthology.parallel:result is None 2024-09-13T07:17:44.292 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T07:17:44.299 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T07:17:44.299 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T07:17:44.307 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T07:17:44.314 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T07:17:44.321 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T07:17:44.321 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T07:17:44.340 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T07:17:44.347 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T07:17:44.347 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T07:17:44.426 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T07:17:44.437 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T07:17:44.439 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T07:17:44.446 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T07:17:44.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T07:17:44.461 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T07:17:44.470 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T07:17:44.470 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T07:17:44.478 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T07:17:44.527 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T07:17:44.533 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:17:44.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T07:17:44.554 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-09-13T07:17:44.648 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T07:17:44.655 INFO:teuthology.task.internal:Starting timer... 2024-09-13T07:17:44.655 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T07:17:44.668 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T07:17:44.677 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2024-09-13T07:17:44.677 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T07:17:44.677 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T07:17:44.677 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T07:17:44.677 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T07:17:44.685 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T07:17:44.697 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T07:17:44.698 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T07:23:18.531 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com')] 2024-09-13T07:23:18.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T07:23:18.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T07:23:18.659 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-13T07:23:18.786 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T07:23:18.786 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T07:23:18.798 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T07:23:18.798 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T07:23:18.799 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T07:23:18.835 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-13T07:23:18.835 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Command line: ntpd -gq 2024-09-13T07:23:18.836 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: proto: precision = 0.123 usec (-23) 2024-09-13T07:23:18.837 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T07:23:18.837 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T07:23:18.837 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T07:23:18.838 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T07:23:18.838 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T07:23:18.838 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T07:23:18.838 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T07:23:18.839 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listen normally on 3 ens1f0 172.21.15.5:123 2024-09-13T07:23:18.839 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listen normally on 4 lo [::1]:123 2024-09-13T07:23:18.839 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%4]:123 2024-09-13T07:23:18.839 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:18 ntpd[13595]: Listening on routing socket on fd #22 for interface updates 2024-09-13T07:23:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:20 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:20 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:20 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:20 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:20.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:20 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:20 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:21.835 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:21 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:21 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:22.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:22 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:22.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:22 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:22.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:22 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:22.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:22 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:22.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:22 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:22.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:22 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:23.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:23.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:23 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:24.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:24 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:24.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:24 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:24.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:24 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:24.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:24 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:24.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:24 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:24.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:24 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:25.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:25.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:25.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:25 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:26.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:26.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:26.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T07:23:26.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T07:23:26.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T07:23:26.836 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T07:23:26.837 INFO:teuthology.orchestra.run.smithi005.stdout:13 Sep 07:23:26 ntpd[13595]: ntpd: time slew +0.003316 s 2024-09-13T07:23:26.837 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew +0.003316s 2024-09-13T07:23:26.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T07:23:26.837 INFO:teuthology.orchestra.run.smithi005.stderr:13 Sep 07:23:26 ntpd[13595]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T07:23:26.891 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T07:23:26.891 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-09-13T07:23:26.891 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T07:23:26.891 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T07:23:26.891 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T07:23:26.891 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T07:23:26.892 INFO:teuthology.run_tasks:Running task install... 2024-09-13T07:23:26.901 DEBUG:teuthology.task.install:project ceph 2024-09-13T07:23:26.901 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}} 2024-09-13T07:23:26.901 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'} 2024-09-13T07:23:26.901 INFO:teuthology.task.install:Using flavor: default 2024-09-13T07:23:26.907 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T07:23:26.907 INFO:teuthology.task.install:extra packages: [] 2024-09-13T07:23:26.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-key list | grep Ceph 2024-09-13T07:23:27.005 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-13T07:23:27.067 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T07:23:27.067 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T07:23:27.068 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T07:23:27.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T07:23:27.228 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default/ 2024-09-13T07:23:27.228 INFO:teuthology.task.install.deb:Package version is 18.2.4-820-g48a7de7e-1focal 2024-09-13T07:23:27.349 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:23:27.349 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T07:23:27.367 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-13T07:23:27.510 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T07:23:27.517 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-13T07:23:27.535 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-13T07:23:27.567 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal InRelease 2024-09-13T07:23:27.589 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-13T07:23:27.612 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal Release.gpg 2024-09-13T07:23:27.627 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-13T07:23:27.633 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main i386 Packages [3,433 B] 2024-09-13T07:23:27.667 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-13T07:23:28.184 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 28.7 kB in 1s (44.5 kB/s) 2024-09-13T07:23:29.388 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T07:23:29.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-820-g48a7de7e-1focal cephadm=18.2.4-820-g48a7de7e-1focal ceph-mds=18.2.4-820-g48a7de7e-1focal ceph-mgr=18.2.4-820-g48a7de7e-1focal ceph-common=18.2.4-820-g48a7de7e-1focal ceph-fuse=18.2.4-820-g48a7de7e-1focal ceph-test=18.2.4-820-g48a7de7e-1focal ceph-volume=18.2.4-820-g48a7de7e-1focal radosgw=18.2.4-820-g48a7de7e-1focal python3-rados=18.2.4-820-g48a7de7e-1focal python3-rgw=18.2.4-820-g48a7de7e-1focal python3-cephfs=18.2.4-820-g48a7de7e-1focal python3-rbd=18.2.4-820-g48a7de7e-1focal libcephfs2=18.2.4-820-g48a7de7e-1focal libcephfs-dev=18.2.4-820-g48a7de7e-1focal librados2=18.2.4-820-g48a7de7e-1focal librbd1=18.2.4-820-g48a7de7e-1focal rbd-fuse=18.2.4-820-g48a7de7e-1focal 2024-09-13T07:23:29.488 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-13T07:23:29.711 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-13T07:23:29.711 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-13T07:23:29.921 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-13T07:23:29.923 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-13T07:23:29.925 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-13T07:23:29.925 INFO:teuthology.orchestra.run.smithi005.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-13T07:23:29.925 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-13T07:23:29.925 INFO:teuthology.orchestra.run.smithi005.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-13T07:23:29.925 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-13T07:23:29.926 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-09-13T07:23:29.927 INFO:teuthology.orchestra.run.smithi005.stdout: btrfs-tools 2024-09-13T07:23:30.051 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-09-13T07:23:30.051 INFO:teuthology.orchestra.run.smithi005.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T07:23:30.051 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T07:23:30.051 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-13T07:23:30.053 INFO:teuthology.orchestra.run.smithi005.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-13T07:23:30.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2024-09-13T07:23:30.055 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 librbd1 2024-09-13T07:23:30.202 INFO:teuthology.orchestra.run.smithi005.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-13T07:23:30.202 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 199 MB of archives. 2024-09-13T07:23:30.202 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-13T07:23:30.202 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-820-g48a7de7e-1focal [3,456 kB] 2024-09-13T07:23:30.248 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-13T07:23:30.373 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-820-g48a7de7e-1focal [4,039 kB] 2024-09-13T07:23:30.431 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-820-g48a7de7e-1focal [746 kB] 2024-09-13T07:23:30.442 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-820-g48a7de7e-1focal [387 kB] 2024-09-13T07:23:30.449 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-820-g48a7de7e-1focal [31.9 kB] 2024-09-13T07:23:30.449 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-820-g48a7de7e-1focal [200 kB] 2024-09-13T07:23:30.452 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-820-g48a7de7e-1focal [62.5 kB] 2024-09-13T07:23:30.453 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-820-g48a7de7e-1focal [367 kB] 2024-09-13T07:23:30.460 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-820-g48a7de7e-1focal [6,628 kB] 2024-09-13T07:23:30.569 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-820-g48a7de7e-1focal [113 kB] 2024-09-13T07:23:30.570 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-820-g48a7de7e-1focal [478 kB] 2024-09-13T07:23:30.577 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-820-g48a7de7e-1focal [25.2 MB] 2024-09-13T07:23:30.650 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-13T07:23:30.663 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-13T07:23:30.747 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-13T07:23:30.992 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-13T07:23:31.068 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-13T07:23:31.099 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-13T07:23:31.102 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-13T07:23:31.102 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-13T07:23:31.105 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-13T07:23:31.105 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-13T07:23:31.112 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-13T07:23:31.112 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-13T07:23:31.114 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-13T07:23:31.155 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-13T07:23:31.163 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-820-g48a7de7e-1focal [9,612 kB] 2024-09-13T07:23:31.196 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-13T07:23:31.197 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-13T07:23:31.199 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T07:23:31.200 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-13T07:23:31.201 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-13T07:23:31.203 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-13T07:23:31.204 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-13T07:23:31.241 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-13T07:23:31.243 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-13T07:23:31.275 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-820-g48a7de7e-1focal [238 kB] 2024-09-13T07:23:31.278 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-820-g48a7de7e-1focal [148 kB] 2024-09-13T07:23:31.281 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-820-g48a7de7e-1focal [1,575 kB] 2024-09-13T07:23:31.307 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-820-g48a7de7e-1focal [6,640 kB] 2024-09-13T07:23:31.321 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-13T07:23:31.330 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-13T07:23:31.331 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-13T07:23:31.332 INFO:teuthology.orchestra.run.smithi005.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-13T07:23:31.332 INFO:teuthology.orchestra.run.smithi005.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-13T07:23:31.333 INFO:teuthology.orchestra.run.smithi005.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-13T07:23:31.335 INFO:teuthology.orchestra.run.smithi005.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-13T07:23:31.403 INFO:teuthology.orchestra.run.smithi005.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-13T07:23:31.408 INFO:teuthology.orchestra.run.smithi005.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-13T07:23:31.409 INFO:teuthology.orchestra.run.smithi005.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-13T07:23:31.416 INFO:teuthology.orchestra.run.smithi005.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-820-g48a7de7e-1focal [24.5 MB] 2024-09-13T07:23:31.484 INFO:teuthology.orchestra.run.smithi005.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-13T07:23:31.487 INFO:teuthology.orchestra.run.smithi005.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-13T07:23:31.488 INFO:teuthology.orchestra.run.smithi005.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-13T07:23:31.489 INFO:teuthology.orchestra.run.smithi005.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-13T07:23:31.490 INFO:teuthology.orchestra.run.smithi005.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-13T07:23:31.492 INFO:teuthology.orchestra.run.smithi005.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-13T07:23:31.493 INFO:teuthology.orchestra.run.smithi005.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-13T07:23:31.566 INFO:teuthology.orchestra.run.smithi005.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-13T07:23:31.566 INFO:teuthology.orchestra.run.smithi005.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-13T07:23:31.570 INFO:teuthology.orchestra.run.smithi005.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T07:23:31.647 INFO:teuthology.orchestra.run.smithi005.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-13T07:23:31.650 INFO:teuthology.orchestra.run.smithi005.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-13T07:23:31.653 INFO:teuthology.orchestra.run.smithi005.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-13T07:23:31.654 INFO:teuthology.orchestra.run.smithi005.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-13T07:23:31.655 INFO:teuthology.orchestra.run.smithi005.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-13T07:23:31.656 INFO:teuthology.orchestra.run.smithi005.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-13T07:23:31.907 INFO:teuthology.orchestra.run.smithi005.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-820-g48a7de7e-1focal [13.8 kB] 2024-09-13T07:23:31.908 INFO:teuthology.orchestra.run.smithi005.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-820-g48a7de7e-1focal [855 kB] 2024-09-13T07:23:31.921 INFO:teuthology.orchestra.run.smithi005.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-820-g48a7de7e-1focal [2,295 kB] 2024-09-13T07:23:31.938 INFO:teuthology.orchestra.run.smithi005.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-820-g48a7de7e-1focal [238 kB] 2024-09-13T07:23:31.939 INFO:teuthology.orchestra.run.smithi005.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-820-g48a7de7e-1focal [143 kB] 2024-09-13T07:23:31.941 INFO:teuthology.orchestra.run.smithi005.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-820-g48a7de7e-1focal [7,015 kB] 2024-09-13T07:23:32.058 INFO:teuthology.orchestra.run.smithi005.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-13T07:23:32.058 INFO:teuthology.orchestra.run.smithi005.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-820-g48a7de7e-1focal [7,868 kB] 2024-09-13T07:23:32.081 INFO:teuthology.orchestra.run.smithi005.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-13T07:23:32.083 INFO:teuthology.orchestra.run.smithi005.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-13T07:23:32.163 INFO:teuthology.orchestra.run.smithi005.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-13T07:23:32.164 INFO:teuthology.orchestra.run.smithi005.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-820-g48a7de7e-1focal [13.9 kB] 2024-09-13T07:23:32.164 INFO:teuthology.orchestra.run.smithi005.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-13T07:23:32.164 INFO:teuthology.orchestra.run.smithi005.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-820-g48a7de7e-1focal [46.9 MB] 2024-09-13T07:23:32.165 INFO:teuthology.orchestra.run.smithi005.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-13T07:23:32.165 INFO:teuthology.orchestra.run.smithi005.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-13T07:23:32.222 INFO:teuthology.orchestra.run.smithi005.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-13T07:23:32.223 INFO:teuthology.orchestra.run.smithi005.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-13T07:23:32.224 INFO:teuthology.orchestra.run.smithi005.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-13T07:23:32.225 INFO:teuthology.orchestra.run.smithi005.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-13T07:23:32.226 INFO:teuthology.orchestra.run.smithi005.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-13T07:23:32.306 INFO:teuthology.orchestra.run.smithi005.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-13T07:23:32.312 INFO:teuthology.orchestra.run.smithi005.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-13T07:23:32.315 INFO:teuthology.orchestra.run.smithi005.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-13T07:23:32.320 INFO:teuthology.orchestra.run.smithi005.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-13T07:23:32.320 INFO:teuthology.orchestra.run.smithi005.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-13T07:23:32.323 INFO:teuthology.orchestra.run.smithi005.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-13T07:23:32.387 INFO:teuthology.orchestra.run.smithi005.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-13T07:23:32.411 INFO:teuthology.orchestra.run.smithi005.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-13T07:23:32.413 INFO:teuthology.orchestra.run.smithi005.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-13T07:23:32.471 INFO:teuthology.orchestra.run.smithi005.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-13T07:23:32.635 INFO:teuthology.orchestra.run.smithi005.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-13T07:23:32.637 INFO:teuthology.orchestra.run.smithi005.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-13T07:23:32.637 INFO:teuthology.orchestra.run.smithi005.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-13T07:23:32.639 INFO:teuthology.orchestra.run.smithi005.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-13T07:23:32.653 INFO:teuthology.orchestra.run.smithi005.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T07:23:32.653 INFO:teuthology.orchestra.run.smithi005.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-13T07:23:32.655 INFO:teuthology.orchestra.run.smithi005.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-13T07:23:32.655 INFO:teuthology.orchestra.run.smithi005.stdout:Get:107 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-13T07:23:32.656 INFO:teuthology.orchestra.run.smithi005.stdout:Get:108 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-13T07:23:33.165 INFO:teuthology.orchestra.run.smithi005.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-820-g48a7de7e-1focal [120 kB] 2024-09-13T07:23:33.167 INFO:teuthology.orchestra.run.smithi005.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-820-g48a7de7e-1focal [37.4 kB] 2024-09-13T07:23:33.169 INFO:teuthology.orchestra.run.smithi005.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-820-g48a7de7e-1focal [13.5 MB] 2024-09-13T07:23:33.402 INFO:teuthology.orchestra.run.smithi005.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/reef/48a7de7e6e1701a88fd571c7e34908ac4b8ed412/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-820-g48a7de7e-1focal [107 kB] 2024-09-13T07:23:34.071 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 199 MB in 3s (59.5 MB/s) 2024-09-13T07:23:34.235 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-13T07:23:34.291 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-13T07:23:34.296 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-13T07:23:34.384 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-13T07:23:34.779 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T07:23:34.795 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-13T07:23:34.837 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-13T07:23:35.124 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T07:23:35.143 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-13T07:23:35.174 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-13T07:23:35.644 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T07:23:35.661 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T07:23:35.712 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T07:23:36.247 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T07:23:36.265 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T07:23:36.305 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T07:23:36.634 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T07:23:36.653 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T07:23:36.683 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T07:23:37.086 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-13T07:23:37.105 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-13T07:23:37.144 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-13T07:23:37.500 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../007-librbd1_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:37.640 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librbd1 (18.2.4-820-g48a7de7e-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-13T07:23:38.354 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../008-librados2_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:38.477 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librados2 (18.2.4-820-g48a7de7e-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-13T07:23:39.200 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T07:23:39.216 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs2 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:39.602 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rados. 2024-09-13T07:23:39.618 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../010-python3-rados_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:39.660 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rados (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:39.988 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T07:23:40.008 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:23:40.063 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-argparse (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:40.310 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T07:23:40.328 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:40.366 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cephfs (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:40.735 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T07:23:40.753 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:23:40.785 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-common (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:41.113 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T07:23:41.130 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-13T07:23:41.171 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-13T07:23:41.458 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T07:23:41.476 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:41.516 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rbd (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:41.885 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-13T07:23:41.902 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-13T07:23:41.943 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-13T07:23:42.296 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-13T07:23:42.314 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-13T07:23:42.369 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-13T07:23:42.725 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T07:23:42.741 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-13T07:23:42.783 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-13T07:23:43.161 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librgw2. 2024-09-13T07:23:43.178 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../019-librgw2_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:43.219 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librgw2 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:44.097 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T07:23:44.112 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:44.189 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rgw (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:44.524 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liboath0. 2024-09-13T07:23:44.540 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-13T07:23:44.591 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-13T07:23:44.935 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T07:23:44.951 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:44.993 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libradosstriper1 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:45.480 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-common. 2024-09-13T07:23:45.496 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../023-ceph-common_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:45.538 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-common (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:48.044 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-base. 2024-09-13T07:23:48.060 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../024-ceph-base_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:48.113 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-base (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:49.307 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T07:23:49.325 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-13T07:23:49.365 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-13T07:23:49.694 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T07:23:49.710 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-13T07:23:49.743 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-13T07:23:50.030 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T07:23:50.046 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-13T07:23:50.109 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-13T07:23:50.425 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T07:23:50.442 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-13T07:23:50.482 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-13T07:23:50.794 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-mako. 2024-09-13T07:23:50.813 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-13T07:23:50.852 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-13T07:23:51.197 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T07:23:51.214 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T07:23:51.255 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T07:23:51.566 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T07:23:51.583 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-13T07:23:51.625 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-13T07:23:51.953 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webob. 2024-09-13T07:23:51.969 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-13T07:23:52.010 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-13T07:23:52.347 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-13T07:23:52.363 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-13T07:23:52.405 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-13T07:23:52.733 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bs4. 2024-09-13T07:23:52.750 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-13T07:23:52.791 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-13T07:23:53.170 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T07:23:53.189 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-13T07:23:53.231 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-13T07:23:53.547 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T07:23:53.568 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-13T07:23:53.606 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-13T07:23:53.951 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-paste. 2024-09-13T07:23:53.967 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-13T07:23:54.008 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-13T07:23:54.370 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T07:23:54.387 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-13T07:23:54.428 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-13T07:23:54.723 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T07:23:54.739 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-13T07:23:54.781 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-13T07:23:55.101 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T07:23:55.117 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-13T07:23:55.159 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-13T07:23:55.470 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T07:23:55.487 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-13T07:23:55.545 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-13T07:23:55.900 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T07:23:55.918 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-13T07:23:55.958 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-13T07:23:56.260 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T07:23:56.276 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:23:56.318 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:56.731 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T07:23:56.748 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:56.789 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:57.109 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T07:23:57.129 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-13T07:23:57.167 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-13T07:23:57.520 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cryptography. 2024-09-13T07:23:57.537 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-13T07:23:57.578 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-13T07:23:57.965 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T07:23:57.982 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-13T07:23:58.023 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-13T07:23:58.369 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T07:23:58.386 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:58.427 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:58.905 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T07:23:58.924 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:58.963 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mon (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:23:59.913 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T07:23:59.930 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:23:59.971 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-osd (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:02.191 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph. 2024-09-13T07:24:02.209 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../051-ceph_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:02.249 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:03.343 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T07:24:03.359 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:03.401 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-fuse (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T07:24:03.828 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:03.895 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mds (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:04.415 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cephadm. 2024-09-13T07:24:04.431 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../054-cephadm_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:04.498 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cephadm (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:04.793 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T07:24:04.809 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-13T07:24:04.851 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-13T07:24:05.196 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T07:24:05.213 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-13T07:24:05.246 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-13T07:24:05.557 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T07:24:05.577 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:24:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:05.977 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jwt. 2024-09-13T07:24:05.996 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-13T07:24:06.027 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-13T07:24:06.330 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-attr. 2024-09-13T07:24:06.349 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-13T07:24:06.388 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-13T07:24:06.716 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T07:24:06.731 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-13T07:24:06.774 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-13T07:24:07.094 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-py. 2024-09-13T07:24:07.113 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-13T07:24:07.151 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-13T07:24:07.496 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-13T07:24:07.514 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-13T07:24:07.554 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-13T07:24:07.807 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-packaging. 2024-09-13T07:24:07.826 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-13T07:24:07.857 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-13T07:24:08.193 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T07:24:08.210 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-13T07:24:08.251 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-13T07:24:08.588 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T07:24:08.604 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-13T07:24:08.662 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-13T07:24:09.025 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T07:24:09.045 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T07:24:09.082 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T07:24:09.369 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-underscore. 2024-09-13T07:24:09.386 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-13T07:24:09.427 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-13T07:24:09.706 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-13T07:24:09.723 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-13T07:24:09.764 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-13T07:24:10.075 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-routes. 2024-09-13T07:24:10.093 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-13T07:24:10.133 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-13T07:24:10.528 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T07:24:10.544 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:24:10.586 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:12.000 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-decorator. 2024-09-13T07:24:12.016 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-13T07:24:12.058 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-13T07:24:12.394 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-13T07:24:12.413 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-13T07:24:12.452 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-13T07:24:12.738 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-scipy. 2024-09-13T07:24:12.757 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-13T07:24:12.796 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-13T07:24:14.114 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-13T07:24:14.131 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-13T07:24:14.171 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-13T07:24:14.701 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T07:24:14.720 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-13T07:24:14.758 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-13T07:24:15.129 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T07:24:15.146 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-13T07:24:15.187 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-13T07:24:15.904 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T07:24:15.924 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:24:15.961 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:17.065 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T07:24:17.083 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-13T07:24:17.123 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-13T07:24:17.426 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T07:24:17.443 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-13T07:24:17.487 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-13T07:24:17.857 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T07:24:17.874 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-13T07:24:17.915 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-13T07:24:18.330 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T07:24:18.349 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-13T07:24:18.388 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-13T07:24:19.050 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T07:24:19.067 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-13T07:24:19.108 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-13T07:24:19.463 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-blinker. 2024-09-13T07:24:19.481 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-13T07:24:19.522 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-13T07:24:19.827 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-13T07:24:19.844 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-13T07:24:19.885 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-13T07:24:20.264 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T07:24:20.284 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-13T07:24:20.322 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-13T07:24:20.633 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T07:24:20.653 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-13T07:24:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-13T07:24:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T07:24:21.030 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-13T07:24:21.069 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-13T07:24:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T07:24:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:24:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T07:24:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-13T07:24:22.211 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-13T07:24:22.543 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T07:24:22.560 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T07:24:22.601 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-13T07:24:22.855 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package jq. 2024-09-13T07:24:22.872 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T07:24:22.913 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-13T07:24:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2024-09-13T07:24:23.218 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-13T07:24:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-13T07:24:23.580 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T07:24:23.598 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-13T07:24:23.637 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-13T07:24:23.959 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-test. 2024-09-13T07:24:23.975 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../094-ceph-test_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:24.025 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-test (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:28.163 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T07:24:28.179 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-820-g48a7de7e-1focal_all.deb ... 2024-09-13T07:24:28.229 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-volume (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:28.541 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-13T07:24:28.559 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-13T07:24:28.599 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-13T07:24:29.120 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T07:24:29.139 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:29.178 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs-dev (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:29.590 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T07:24:29.609 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-13T07:24:29.665 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-13T07:24:29.977 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T07:24:29.995 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-13T07:24:30.035 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-13T07:24:30.423 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-13T07:24:30.444 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-13T07:24:30.480 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-13T07:24:32.184 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-13T07:24:32.203 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-13T07:24:32.242 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-13T07:24:32.653 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webencodings. 2024-09-13T07:24:32.671 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-13T07:24:32.711 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-13T07:24:33.023 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-html5lib. 2024-09-13T07:24:33.040 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-13T07:24:33.106 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-13T07:24:33.451 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-13T07:24:33.468 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-13T07:24:33.509 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-13T07:24:33.979 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T07:24:33.996 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T07:24:34.037 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T07:24:34.382 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T07:24:34.399 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-13T07:24:34.440 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-13T07:24:34.768 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T07:24:34.785 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-13T07:24:34.826 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-13T07:24:35.146 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T07:24:35.163 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-13T07:24:35.203 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-13T07:24:35.490 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T07:24:35.507 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-13T07:24:35.556 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-13T07:24:36.213 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package radosgw. 2024-09-13T07:24:36.234 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../110-radosgw_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:36.271 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking radosgw (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:37.691 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T07:24:37.710 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-820-g48a7de7e-1focal_amd64.deb ... 2024-09-13T07:24:37.749 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking rbd-fuse (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:38.086 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-13T07:24:38.387 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-13T07:24:38.522 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-13T07:24:38.640 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-13T07:24:38.824 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cephadm (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:39.395 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user cephadm....done 2024-09-13T07:24:39.550 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-13T07:24:39.701 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-13T07:24:40.053 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T07:24:40.329 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-13T07:24:40.455 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-13T07:24:40.581 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-13T07:24:40.916 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-13T07:24:41.311 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-13T07:24:41.596 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-13T07:24:41.889 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-13T07:24:42.165 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-13T07:24:42.441 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-argparse (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:42.729 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-13T07:24:42.851 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-13T07:24:43.127 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-13T07:24:43.420 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-13T07:24:43.554 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-13T07:24:43.931 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-13T07:24:44.232 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-13T07:24:44.508 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-13T07:24:44.842 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-13T07:24:45.135 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-13T07:24:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-13T07:24:45.771 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-13T07:24:46.088 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-13T07:24:46.239 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-13T07:24:46.366 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-13T07:24:46.650 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-13T07:24:47.043 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-13T07:24:47.428 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-13T07:24:47.813 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-13T07:24:47.939 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-common (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:24:48.276 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T07:24:48.410 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-13T07:24:48.762 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-13T07:24:48.890 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-13T07:24:49.013 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-13T07:24:49.306 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-13T07:24:49.863 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T07:24:50.258 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-13T07:24:50.385 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-13T07:24:50.736 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-13T07:24:51.012 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-13T07:24:51.330 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-13T07:24:51.456 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T07:24:51.732 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-13T07:24:52.025 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-13T07:24:52.334 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-13T07:24:52.684 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-13T07:24:52.937 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-13T07:24:53.063 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-13T07:24:53.389 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-13T07:24:53.774 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-13T07:24:53.899 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-13T07:24:54.045 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-13T07:24:54.377 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-13T07:24:54.695 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-13T07:24:56.200 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-13T07:24:56.335 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-13T07:24:56.736 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-13T07:24:57.054 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-13T07:24:57.347 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T07:24:57.473 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-13T07:24:57.824 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-13T07:24:58.142 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-13T07:24:58.552 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-13T07:24:58.878 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-13T07:24:59.933 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-13T07:25:00.093 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-13T07:25:00.219 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-13T07:25:00.345 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-13T07:25:00.646 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-13T07:25:00.772 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-13T07:25:01.081 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-13T07:25:01.399 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T07:25:01.684 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-13T07:25:02.070 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librados2 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:02.196 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-13T07:25:02.322 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librgw2 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:02.473 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libsqlite3-mod-ceph (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:02.692 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs2 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:02.818 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-13T07:25:03.128 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libradosstriper1 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:03.254 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-13T07:25:03.581 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librbd1 (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:03.707 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-modules-core (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:03.833 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-fuse (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:04.061 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T07:25:04.061 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T07:25:04.501 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs-dev (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:04.619 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rados (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:04.770 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-13T07:25:05.707 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rbd (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:05.833 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up rbd-fuse (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:05.993 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rgw (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:06.110 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cephfs (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:06.228 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-common (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:06.806 INFO:teuthology.orchestra.run.smithi005.stdout:Adding group ceph....done 2024-09-13T07:25:07.200 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user ceph....done 2024-09-13T07:25:07.367 INFO:teuthology.orchestra.run.smithi005.stdout:Setting system user ceph properties....done 2024-09-13T07:25:07.391 INFO:teuthology.orchestra.run.smithi005.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T07:25:07.543 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T07:25:07.865 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T07:25:08.346 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-test (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:08.472 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up radosgw (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:08.944 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T07:25:08.944 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T07:25:09.383 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-base (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:09.795 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T07:25:10.268 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mds (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:10.522 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T07:25:10.522 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T07:25:10.986 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:11.260 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T07:25:11.260 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T07:25:11.964 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-osd (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:12.483 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T07:25:12.483 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T07:25:12.932 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-k8sevents (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:13.199 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:13.351 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mon (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:13.583 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T07:25:13.583 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T07:25:14.036 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-cephadm (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:14.178 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:14.304 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-dashboard (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:14.472 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-volume (18.2.4-820-g48a7de7e-1focal) ... 2024-09-13T07:25:14.833 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-13T07:25:15.019 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-13T07:25:15.303 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-13T07:25:18.198 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-13T07:25:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T07:25:18.799 DEBUG:teuthology.parallel:result is None 2024-09-13T07:25:18.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=48a7de7e6e1701a88fd571c7e34908ac4b8ed412 2024-09-13T07:25:18.956 DEBUG:teuthology.orchestra.run.smithi005:> dpkg-query -W -f '${Version}' ceph 2024-09-13T07:25:18.980 INFO:teuthology.orchestra.run.smithi005.stdout:18.2.4-820-g48a7de7e-1focal 2024-09-13T07:25:18.981 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-820-g48a7de7e-1focal 2024-09-13T07:25:18.981 INFO:teuthology.task.install:The correct ceph version 18.2.4-820-g48a7de7e-1focal is installed. 2024-09-13T07:25:18.982 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T07:25:18.983 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:18.983 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T07:25:19.037 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T07:25:19.038 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:19.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T07:25:19.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T07:25:19.062 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T07:25:19.063 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:19.063 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T07:25:19.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T07:25:19.128 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T07:25:19.128 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:19.128 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T07:25:19.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T07:25:19.201 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T07:25:19.296 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T07:25:19.296 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-09-13T07:25:19.312 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T07:25:19.312 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T07:25:19.372 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T07:25:19.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T07:25:19.391 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T07:25:19.391 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 125}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T07:25:19.392 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930', 'branch': 'reef', 'description': 'rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_20.04} workloads/rbd_cli_generic}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900930', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 125}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 248, 'sha1': '48a7de7e6e1701a88fd571c7e34908ac4b8ed412', 'sleep_before_teardown': 0, 'subset': '116/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7a853b1706d076d58e0da7a5d3b9b9eb36276880/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7a853b1706d076d58e0da7a5d3b9b9eb36276880', 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'workunit': {'clients': {'client.0': ['rbd/cli_generic.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_22:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T07:25:19.392 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T07:25:19.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T07:25:19.455 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:19.455 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T07:25:19.461 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T07:25:19.461 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 358 Links: 1 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 07:21:53.538906673 +0000 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 07:21:53.346911090 +0000 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 07:21:53.346911090 +0000 2024-09-13T07:25:19.466 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T07:25:19.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T07:25:19.477 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T07:25:19.477 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T07:25:19.477 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00030661 s, 1.7 MB/s 2024-09-13T07:25:19.479 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T07:25:19.486 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 07:21:53.970896732 +0000 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 07:21:53.762901518 +0000 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 07:21:53.762901518 +0000 2024-09-13T07:25:19.534 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T07:25:19.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T07:25:19.546 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T07:25:19.547 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T07:25:19.547 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000313333 s, 1.6 MB/s 2024-09-13T07:25:19.548 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T07:25:19.556 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-09-13T07:25:19.563 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 372 Links: 1 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 07:21:54.426886241 +0000 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 07:21:54.186891763 +0000 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 07:21:54.186891763 +0000 2024-09-13T07:25:19.564 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T07:25:19.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T07:25:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T07:25:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T07:25:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00029192 s, 1.8 MB/s 2024-09-13T07:25:19.623 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T07:25:19.630 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-09-13T07:25:19.636 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T07:25:19.636 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T07:25:19.636 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-13T07:25:19.636 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T07:25:19.637 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 07:21:54.858876297 +0000 2024-09-13T07:25:19.637 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 07:21:54.626881637 +0000 2024-09-13T07:25:19.637 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 07:21:54.626881637 +0000 2024-09-13T07:25:19.637 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-13T07:25:19.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T07:25:19.647 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T07:25:19.647 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T07:25:19.648 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00028977 s, 1.8 MB/s 2024-09-13T07:25:19.649 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T07:25:19.658 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T07:25:19.658 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-13T07:25:19.658 INFO:tasks.ceph:Generating config... 2024-09-13T07:25:19.659 INFO:tasks.ceph:[client] rbd default data pool = datapool 2024-09-13T07:25:19.659 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-13T07:25:19.659 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T07:25:19.659 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-13T07:25:19.659 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T07:25:19.659 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T07:25:19.660 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T07:25:19.660 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T07:25:19.660 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T07:25:19.660 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-13T07:25:19.660 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T07:25:19.661 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T07:25:19.661 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T07:25:19.661 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T07:25:19.661 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T07:25:19.661 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-13T07:25:19.661 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T07:25:19.661 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T07:25:19.698 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T07:25:19.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T07:25:19.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T07:25:19.806 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.5')] 2024-09-13T07:25:19.806 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 125}, 'mon.a': {}} 2024-09-13T07:25:19.807 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:19.807 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T07:25:19.813 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid 9dfd7139-cdcf-49aa-a58a-f69fdac1ee97 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = pacific 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 9dfd7139-cdcf-49aa-a58a-f69fdac1ee97 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-09-13T07:25:19.854387+0000 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-13T07:25:19.854387+0000 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 16 (pacific) 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T07:25:19.851 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-13T07:25:19.853 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T07:25:19.857 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9dfd7139-cdcf-49aa-a58a-f69fdac1ee97... 2024-09-13T07:25:19.858 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T07:25:19.892 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-09-13T07:25:19.892 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-09-13T07:25:19.892 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T07:25:19.892 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-09-13T07:25:19.893 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-09-13T07:25:19.894 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-09-13T07:25:19.895 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = 172.21.15.5 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on pool no app = False 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 5000 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 9dfd7139-cdcf-49aa-a58a-f69fdac1ee97 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-09-13T07:25:19.896 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.897 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression algorithm = lz4 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression mode = aggressive 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 1/20 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 1/20 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 4/10 2024-09-13T07:25:19.898 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: osd shutdown pgref assert = True 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-09-13T07:25:19.899 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T07:25:19.900 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-09-13T07:25:19.901 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-09-13T07:25:19.902 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T07:25:19.902 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T07:25:19.902 INFO:teuthology.orchestra.run.smithi005.stdout: rbd default data pool = datapool 2024-09-13T07:25:19.902 INFO:teuthology.orchestra.run.smithi005.stdout: rbd default features = 125 2024-09-13T07:25:19.902 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-09-13T07:25:19.903 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T07:25:19.903 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T07:25:20.002 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T07:25:20.002 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:20.002 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T07:25:20.009 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:20.009 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T07:25:20.016 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:25:20.016 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:20.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T07:25:20.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T07:25:20.084 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:20.084 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:20.090 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T07:25:20.090 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T07:25:20.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T07:25:20.137 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T07:25:20.140 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T07:25:20.140 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T07:25:20.141 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T07:25:20.225 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T07:25:20.239 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T07:25:20.239 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-13T07:25:20.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T07:25:20.251 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T07:25:20.251 INFO:tasks.ceph:role: osd.0 2024-09-13T07:25:20.251 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:25:20.251 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T07:25:20.292 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T07:25:20.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T07:25:20.294 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T07:25:20.356 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T07:25:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T07:25:20.365 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T07:25:20.366 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T07:25:20.380 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T07:25:20.380 INFO:tasks.ceph:role: osd.1 2024-09-13T07:25:20.381 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:25:20.381 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T07:25:20.466 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T07:25:20.467 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T07:25:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T07:25:20.469 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T07:25:20.469 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T07:25:20.487 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T07:25:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T07:25:20.496 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T07:25:20.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T07:25:20.508 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T07:25:20.508 INFO:tasks.ceph:role: osd.2 2024-09-13T07:25:20.508 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:25:20.508 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T07:25:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T07:25:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T07:25:20.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T07:25:20.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T07:25:20.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T07:25:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T07:25:20.623 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T07:25:20.623 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:20.753+0000 7fdd38728880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T07:25:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:20.753+0000 7fdd38728880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T07:25:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:20.753+0000 7fdd38728880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T07:25:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:20.753+0000 7fdd38728880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T07:25:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:20.753+0000 7fdd38728880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T07:25:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:20.757+0000 7fdd38728880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T07:25:23.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T07:25:23.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:23.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:23.805+0000 7f66aa41c880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T07:25:23.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:23.809+0000 7f66aa41c880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T07:25:23.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:23.809+0000 7f66aa41c880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T07:25:23.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:23.809+0000 7f66aa41c880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T07:25:23.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:23.809+0000 7f66aa41c880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T07:25:23.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:23.809+0000 7f66aa41c880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T07:25:26.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T07:25:26.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:26.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:26.901+0000 7f44f6909880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T07:25:26.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:26.901+0000 7f44f6909880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T07:25:26.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:26.901+0000 7f44f6909880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T07:25:26.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:26.901+0000 7f44f6909880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T07:25:26.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:26.901+0000 7f44f6909880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T07:25:26.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T07:25:26.901+0000 7f44f6909880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T07:25:29.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T07:25:29.910 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T07:25:29.911 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:29.911 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T07:25:29.925 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:29.925 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T07:25:29.939 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:29.939 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T07:25:29.996 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:29.996 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T07:25:30.011 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:30.011 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T07:25:30.018 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T07:25:30.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T07:25:30.028 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-09-13T07:25:30.028 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDg6ONm8RdsCBAAO3Q/JMTLuTQ1vnog8sPe3Q== 2024-09-13T07:25:30.028 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-09-13T07:25:30.028 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDg6ONme+krLRAA1+RwFmw+8vNaPsUp5gno8A== 2024-09-13T07:25:30.029 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-09-13T07:25:30.029 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDj6ONm1iJUMBAAWW6LfNyjj7H84W5t0oJ4ug== 2024-09-13T07:25:30.029 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-09-13T07:25:30.029 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDm6ONmlhvmNRAAAoGHREDijN1NSixyDQ0QGQ== 2024-09-13T07:25:30.029 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-09-13T07:25:30.029 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDg6ONm1FmoDRAAF7LGpmMZgF1oEyv7ioNIWA== 2024-09-13T07:25:30.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T07:25:30.120 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T07:25:30.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T07:25:30.212 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T07:25:30.253 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T07:25:30.303 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T07:25:30.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T07:25:30.318 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T07:25:30.417 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T07:25:30.434 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T07:25:30.438 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T07:25:30.439 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T07:25:30.439 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T07:25:30.485 INFO:tasks.ceph.mon.a:Started 2024-09-13T07:25:30.485 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T07:25:30.485 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T07:25:30.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T07:25:30.487 INFO:tasks.ceph.mgr.x:Started 2024-09-13T07:25:30.488 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T07:25:30.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T07:25:30.738 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-09-13T07:25:30.750 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T07:25:30.750 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T07:25:30.751 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:30.751 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T07:25:30.760 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:30.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T07:25:30.818 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:25:30.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T07:25:30.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 7f078194-fd93-489d-9396-3ca5def6fedb 0 2024-09-13T07:25:30.855 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:30.853+0000 7fbad268f2c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T07:25:31.029 INFO:teuthology.orchestra.run.smithi005.stdout:0 2024-09-13T07:25:31.042 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa 1 2024-09-13T07:25:31.291 INFO:teuthology.orchestra.run.smithi005.stdout:1 2024-09-13T07:25:31.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd new 9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2 2 2024-09-13T07:25:31.511 INFO:teuthology.orchestra.run.smithi005.stdout:2 2024-09-13T07:25:31.524 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T07:25:31.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T07:25:31.527 INFO:tasks.ceph.osd.0:Started 2024-09-13T07:25:31.527 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T07:25:31.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T07:25:31.529 INFO:tasks.ceph.osd.1:Started 2024-09-13T07:25:31.529 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T07:25:31.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T07:25:31.532 INFO:tasks.ceph.osd.2:Started 2024-09-13T07:25:31.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T07:25:31.717 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:31.717+0000 7fbad268f2c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T07:25:31.753 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:31.753 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:25:31.512317+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:25:31.766 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T07:25:31.766 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T07:25:31.781 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:31.781+0000 7fbad268f2c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T07:25:31.909 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T07:25:31.909+0000 7f4b31d32880 -1 Falling back to public interface 2024-09-13T07:25:31.958 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:31.957+0000 7fbad268f2c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T07:25:32.069 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T07:25:32.133 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:25:32.133+0000 7ff3a5789880 -1 Falling back to public interface 2024-09-13T07:25:32.145 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:25:32.145+0000 7f329707d880 -1 Falling back to public interface 2024-09-13T07:25:32.202 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:32.201+0000 7fbad268f2c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T07:25:32.259 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:32.257+0000 7fbad268f2c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T07:25:32.375 INFO:teuthology.misc.health.smithi005.stderr:2024-09-13T07:25:32.377+0000 7f38d67fc700 0 --2- 172.21.15.5:0/285409560 >> v2:172.21.15.5:3300/0 conn(0x7f38e0127260 0x7f38e012c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:25:32.452 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:32.453+0000 7fbad268f2c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T07:25:32.458 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T07:25:32.458 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":5,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:25:31.512317+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:25:32.470 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-13T07:25:32.520 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:32.521+0000 7fbad268f2c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T07:25:32.592 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:32.593+0000 7fbad268f2c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T07:25:33.032 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:33.033+0000 7fbad268f2c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T07:25:33.160 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:33.161+0000 7fbad268f2c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T07:25:33.533 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:33.533+0000 7fbad268f2c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T07:25:33.730 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:33.729+0000 7fbad268f2c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T07:25:33.799 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:33.801+0000 7fbad268f2c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T07:25:33.964 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:33.965+0000 7fbad268f2c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T07:25:34.668 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:34.669+0000 7fbad268f2c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T07:25:34.739 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:34.741+0000 7fbad268f2c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T07:25:34.903 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:34.905+0000 7fbad268f2c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T07:25:34.970 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:34.969+0000 7fbad268f2c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T07:25:35.045 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:35.045+0000 7fbad268f2c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T07:25:35.355 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:35.353+0000 7fbad268f2c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T07:25:35.416 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T07:25:35.417+0000 7f4b31d32880 -1 osd.2 0 log_to_monitors true 2024-09-13T07:25:35.427 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:35.429+0000 7fbad268f2c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T07:25:35.433 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:25:35.433+0000 7f329707d880 -1 osd.1 0 log_to_monitors true 2024-09-13T07:25:35.625 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:25:35.625+0000 7ff3a5789880 -1 osd.0 0 log_to_monitors true 2024-09-13T07:25:36.144 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:36.145+0000 7fbad268f2c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T07:25:37.179 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:25:37.177+0000 7fbab9203700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T07:25:37.675 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T07:25:37.673+0000 7f4b2db71700 -1 osd.2 0 waiting for initial osdmap 2024-09-13T07:25:37.681 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-13T07:25:37.681+0000 7f4b25620700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T07:25:37.743 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:25:37.745+0000 7f3293843700 -1 osd.1 0 waiting for initial osdmap 2024-09-13T07:25:37.749 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:25:37.749+0000 7f328a96b700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T07:25:38.772 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T07:25:38.839 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:25:38.837+0000 7ff3a15c8700 -1 osd.0 0 waiting for initial osdmap 2024-09-13T07:25:38.843 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:25:38.845+0000 7ff399077700 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T07:25:39.063 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T07:25:39.064 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":9,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:25:38.679487+0000","last_up_change":"2024-09-13T07:25:38.679487+0000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6817","nonce":4236811925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6819","nonce":4236811925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6823","nonce":4236811925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6821","nonce":4236811925}]},"public_addr":"172.21.15.5:6817/4236811925","cluster_addr":"172.21.15.5:6819/4236811925","heartbeat_back_addr":"172.21.15.5:6823/4236811925","heartbeat_front_addr":"172.21.15.5:6821/4236811925","state":["exists","up"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6801","nonce":440298088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6803","nonce":440298088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6807","nonce":440298088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6805","nonce":440298088}]},"public_addr":"172.21.15.5:6801/440298088","cluster_addr":"172.21.15.5:6803/440298088","heartbeat_back_addr":"172.21.15.5:6807/440298088","heartbeat_front_addr":"172.21.15.5:6805/440298088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:25:39.076 DEBUG:teuthology.misc:2 of 3 OSDs are up 2024-09-13T07:25:45.379 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T07:25:45.680 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T07:25:45.681 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":13,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:25:42.850751+0000","last_up_change":"2024-09-13T07:25:39.843856+0000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T07:25:40.241777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6809","nonce":1121634097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6811","nonce":1121634097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6815","nonce":1121634097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6813","nonce":1121634097}]},"public_addr":"172.21.15.5:6809/1121634097","cluster_addr":"172.21.15.5:6811/1121634097","heartbeat_back_addr":"172.21.15.5:6815/1121634097","heartbeat_front_addr":"172.21.15.5:6813/1121634097","state":["exists","up"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6817","nonce":4236811925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6819","nonce":4236811925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6823","nonce":4236811925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6821","nonce":4236811925}]},"public_addr":"172.21.15.5:6817/4236811925","cluster_addr":"172.21.15.5:6819/4236811925","heartbeat_back_addr":"172.21.15.5:6823/4236811925","heartbeat_front_addr":"172.21.15.5:6821/4236811925","state":["exists","up"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6801","nonce":440298088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6803","nonce":440298088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6807","nonce":440298088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6805","nonce":440298088}]},"public_addr":"172.21.15.5:6801/440298088","cluster_addr":"172.21.15.5:6803/440298088","heartbeat_back_addr":"172.21.15.5:6807/440298088","heartbeat_front_addr":"172.21.15.5:6805/440298088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.660118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.468047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.373702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:25:45.693 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-13T07:25:45.693 INFO:tasks.ceph:Creating RBD pool 2024-09-13T07:25:45.694 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T07:25:46.879 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2024-09-13T07:25:46.892 DEBUG:teuthology.orchestra.run.smithi005:> rbd --cluster ceph pool init rbd 2024-09-13T07:25:49.893 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T07:25:49.893 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T07:25:49.894 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T07:25:49.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T07:25:49.894 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T07:25:50.243 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:50.255 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":3534510485},{"type":"v1","addr":"172.21.15.5:6825","nonce":3534510485}]},"active_addr":"172.21.15.5:6825/3534510485","active_change":"2024-09-13T07:25:36.168639+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1774977293}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":429749739}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3795888129}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1320595655}]}]} 2024-09-13T07:25:50.257 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T07:25:50.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T07:25:50.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T07:25:50.565 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:50.565 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:25:49.879562+0000","last_up_change":"2024-09-13T07:25:39.843856+0000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T07:25:40.241777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T07:25:46.036455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6809","nonce":1121634097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6811","nonce":1121634097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6815","nonce":1121634097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6813","nonce":1121634097}]},"public_addr":"172.21.15.5:6809/1121634097","cluster_addr":"172.21.15.5:6811/1121634097","heartbeat_back_addr":"172.21.15.5:6815/1121634097","heartbeat_front_addr":"172.21.15.5:6813/1121634097","state":["exists","up"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6817","nonce":4236811925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6819","nonce":4236811925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6823","nonce":4236811925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6821","nonce":4236811925}]},"public_addr":"172.21.15.5:6817/4236811925","cluster_addr":"172.21.15.5:6819/4236811925","heartbeat_back_addr":"172.21.15.5:6823/4236811925","heartbeat_front_addr":"172.21.15.5:6821/4236811925","state":["exists","up"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6801","nonce":440298088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6803","nonce":440298088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6807","nonce":440298088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6805","nonce":440298088}]},"public_addr":"172.21.15.5:6801/440298088","cluster_addr":"172.21.15.5:6803/440298088","heartbeat_back_addr":"172.21.15.5:6807/440298088","heartbeat_front_addr":"172.21.15.5:6805/440298088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.660118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.468047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.373702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:25:50.580 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T07:25:50.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T07:25:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:50.885 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:25:49.879562+0000","last_up_change":"2024-09-13T07:25:39.843856+0000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T07:25:40.241777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T07:25:46.036455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6809","nonce":1121634097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6811","nonce":1121634097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6815","nonce":1121634097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6813","nonce":1121634097}]},"public_addr":"172.21.15.5:6809/1121634097","cluster_addr":"172.21.15.5:6811/1121634097","heartbeat_back_addr":"172.21.15.5:6815/1121634097","heartbeat_front_addr":"172.21.15.5:6813/1121634097","state":["exists","up"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6817","nonce":4236811925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6819","nonce":4236811925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6823","nonce":4236811925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6821","nonce":4236811925}]},"public_addr":"172.21.15.5:6817/4236811925","cluster_addr":"172.21.15.5:6819/4236811925","heartbeat_back_addr":"172.21.15.5:6823/4236811925","heartbeat_front_addr":"172.21.15.5:6821/4236811925","state":["exists","up"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6801","nonce":440298088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6803","nonce":440298088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6807","nonce":440298088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6805","nonce":440298088}]},"public_addr":"172.21.15.5:6801/440298088","cluster_addr":"172.21.15.5:6803/440298088","heartbeat_back_addr":"172.21.15.5:6807/440298088","heartbeat_front_addr":"172.21.15.5:6805/440298088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.660118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.468047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.373702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:25:50.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T07:25:50.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T07:25:50.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T07:25:51.046 INFO:teuthology.orchestra.run.smithi005.stdout:38654705668 2024-09-13T07:25:51.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T07:25:51.049 INFO:teuthology.orchestra.run.smithi005.stdout:42949672966 2024-09-13T07:25:51.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T07:25:51.061 INFO:teuthology.orchestra.run.smithi005.stdout:38654705669 2024-09-13T07:25:51.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T07:25:51.360 INFO:teuthology.orchestra.run.smithi005.stdout:38654705667 2024-09-13T07:25:51.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2024-09-13T07:25:51.399 INFO:teuthology.orchestra.run.smithi005.stdout:42949672964 2024-09-13T07:25:51.407 INFO:teuthology.orchestra.run.smithi005.stdout:38654705668 2024-09-13T07:25:51.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-13T07:25:51.422 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.2 2024-09-13T07:25:52.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T07:25:52.414 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T07:25:52.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T07:25:52.706 INFO:teuthology.orchestra.run.smithi005.stdout:38654705667 2024-09-13T07:25:52.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2024-09-13T07:25:52.755 INFO:teuthology.orchestra.run.smithi005.stdout:42949672964 2024-09-13T07:25:52.769 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-13T07:25:52.782 INFO:teuthology.orchestra.run.smithi005.stdout:38654705668 2024-09-13T07:25:52.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.2 2024-09-13T07:25:53.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T07:25:53.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T07:25:53.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T07:25:54.051 INFO:teuthology.orchestra.run.smithi005.stdout:38654705668 2024-09-13T07:25:54.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2024-09-13T07:25:54.069 DEBUG:teuthology.parallel:result is None 2024-09-13T07:25:54.079 INFO:teuthology.orchestra.run.smithi005.stdout:42949672966 2024-09-13T07:25:54.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.0 2024-09-13T07:25:54.093 DEBUG:teuthology.parallel:result is None 2024-09-13T07:25:54.124 INFO:teuthology.orchestra.run.smithi005.stdout:38654705669 2024-09-13T07:25:54.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2024-09-13T07:25:54.138 DEBUG:teuthology.parallel:result is None 2024-09-13T07:25:54.138 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T07:25:54.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:25:54.446 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:54.448 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:25:54.459 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T07:25:54.180806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93908,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":93307,"kb_avail":283021612,"statfs":{"total":289910292480,"available":289814130688,"internally_reserved":0,"allocated":557056,"data_stored":1305042,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":95546706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.305441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.891797+0000","last_change":"2024-09-13T07:25:49.892092+0000","last_active":"2024-09-13T07:25:49.891797+0000","last_peered":"2024-09-13T07:25:49.891797+0000","last_clean":"2024-09-13T07:25:49.891797+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:25:49.891797+0000","last_undegraded":"2024-09-13T07:25:49.891797+0000","last_fullsized":"2024-09-13T07:25:49.891797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:18.745059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.891461+0000","last_change":"2024-09-13T07:25:49.891916+0000","last_active":"2024-09-13T07:25:49.891461+0000","last_peered":"2024-09-13T07:25:49.891461+0000","last_clean":"2024-09-13T07:25:49.891461+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:25:49.891461+0000","last_undegraded":"2024-09-13T07:25:49.891461+0000","last_fullsized":"2024-09-13T07:25:49.891461+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:09.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.891333+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:25:49.891333+0000","last_peered":"2024-09-13T07:25:49.891333+0000","last_clean":"2024-09-13T07:25:49.891333+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:25:49.891333+0000","last_undegraded":"2024-09-13T07:25:49.891333+0000","last_fullsized":"2024-09-13T07:25:49.891333+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:45.692840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.890787+0000","last_change":"2024-09-13T07:25:49.891182+0000","last_active":"2024-09-13T07:25:49.890787+0000","last_peered":"2024-09-13T07:25:49.890787+0000","last_clean":"2024-09-13T07:25:49.890787+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:25:49.890787+0000","last_undegraded":"2024-09-13T07:25:49.890787+0000","last_fullsized":"2024-09-13T07:25:49.890787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:57:22.870708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.889567+0000","last_change":"2024-09-13T07:25:49.885232+0000","last_active":"2024-09-13T07:25:49.889567+0000","last_peered":"2024-09-13T07:25:49.889567+0000","last_clean":"2024-09-13T07:25:49.889567+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:25:49.889567+0000","last_undegraded":"2024-09-13T07:25:49.889567+0000","last_fullsized":"2024-09-13T07:25:49.889567+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:00.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:50.530387+0000","last_change":"2024-09-13T07:25:50.530745+0000","last_active":"2024-09-13T07:25:50.530387+0000","last_peered":"2024-09-13T07:25:50.530387+0000","last_clean":"2024-09-13T07:25:50.530387+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:25:50.530387+0000","last_undegraded":"2024-09-13T07:25:50.530387+0000","last_fullsized":"2024-09-13T07:25:50.530387+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:25.418612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:50.530441+0000","last_change":"2024-09-13T07:25:50.530802+0000","last_active":"2024-09-13T07:25:50.530441+0000","last_peered":"2024-09-13T07:25:50.530441+0000","last_clean":"2024-09-13T07:25:50.530441+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:25:50.530441+0000","last_undegraded":"2024-09-13T07:25:50.530441+0000","last_fullsized":"2024-09-13T07:25:50.530441+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:58:27.785140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.890916+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:25:49.890916+0000","last_peered":"2024-09-13T07:25:49.890916+0000","last_clean":"2024-09-13T07:25:49.890916+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:25:49.890916+0000","last_undegraded":"2024-09-13T07:25:49.890916+0000","last_fullsized":"2024-09-13T07:25:49.890916+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:42.260372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'97","reported_seq":141,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T07:25:48.883166+0000","last_change":"2024-09-13T07:25:42.741998+0000","last_active":"2024-09-13T07:25:48.883166+0000","last_peered":"2024-09-13T07:25:48.883166+0000","last_clean":"2024-09-13T07:25:48.883166+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:25:48.883166+0000","last_undegraded":"2024-09-13T07:25:48.883166+0000","last_fullsized":"2024-09-13T07:25:48.883166+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_clean_scrub_stamp":"2024-09-13T07:25:40.845594+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:10.726746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":208896,"data_stored":632997,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31260,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340580,"statfs":{"total":96636764160,"available":96604753920,"internally_reserved":0,"allocated":139264,"data_stored":39048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":208896,"data_stored":632997,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:25:54.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:25:54.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:54.769 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:25:54.780 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T07:25:54.180806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93908,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":93307,"kb_avail":283021612,"statfs":{"total":289910292480,"available":289814130688,"internally_reserved":0,"allocated":557056,"data_stored":1305042,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":95546706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.305441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.891797+0000","last_change":"2024-09-13T07:25:49.892092+0000","last_active":"2024-09-13T07:25:49.891797+0000","last_peered":"2024-09-13T07:25:49.891797+0000","last_clean":"2024-09-13T07:25:49.891797+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:25:49.891797+0000","last_undegraded":"2024-09-13T07:25:49.891797+0000","last_fullsized":"2024-09-13T07:25:49.891797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:18.745059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.891461+0000","last_change":"2024-09-13T07:25:49.891916+0000","last_active":"2024-09-13T07:25:49.891461+0000","last_peered":"2024-09-13T07:25:49.891461+0000","last_clean":"2024-09-13T07:25:49.891461+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:25:49.891461+0000","last_undegraded":"2024-09-13T07:25:49.891461+0000","last_fullsized":"2024-09-13T07:25:49.891461+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:09.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.891333+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:25:49.891333+0000","last_peered":"2024-09-13T07:25:49.891333+0000","last_clean":"2024-09-13T07:25:49.891333+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:25:49.891333+0000","last_undegraded":"2024-09-13T07:25:49.891333+0000","last_fullsized":"2024-09-13T07:25:49.891333+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:45.692840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.890787+0000","last_change":"2024-09-13T07:25:49.891182+0000","last_active":"2024-09-13T07:25:49.890787+0000","last_peered":"2024-09-13T07:25:49.890787+0000","last_clean":"2024-09-13T07:25:49.890787+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:25:49.890787+0000","last_undegraded":"2024-09-13T07:25:49.890787+0000","last_fullsized":"2024-09-13T07:25:49.890787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:57:22.870708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.889567+0000","last_change":"2024-09-13T07:25:49.885232+0000","last_active":"2024-09-13T07:25:49.889567+0000","last_peered":"2024-09-13T07:25:49.889567+0000","last_clean":"2024-09-13T07:25:49.889567+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:25:49.889567+0000","last_undegraded":"2024-09-13T07:25:49.889567+0000","last_fullsized":"2024-09-13T07:25:49.889567+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:00.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:50.530387+0000","last_change":"2024-09-13T07:25:50.530745+0000","last_active":"2024-09-13T07:25:50.530387+0000","last_peered":"2024-09-13T07:25:50.530387+0000","last_clean":"2024-09-13T07:25:50.530387+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:25:50.530387+0000","last_undegraded":"2024-09-13T07:25:50.530387+0000","last_fullsized":"2024-09-13T07:25:50.530387+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:25.418612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:50.530441+0000","last_change":"2024-09-13T07:25:50.530802+0000","last_active":"2024-09-13T07:25:50.530441+0000","last_peered":"2024-09-13T07:25:50.530441+0000","last_clean":"2024-09-13T07:25:50.530441+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:25:50.530441+0000","last_undegraded":"2024-09-13T07:25:50.530441+0000","last_fullsized":"2024-09-13T07:25:50.530441+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:58:27.785140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-13T07:25:49.890916+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:25:49.890916+0000","last_peered":"2024-09-13T07:25:49.890916+0000","last_clean":"2024-09-13T07:25:49.890916+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:25:49.890916+0000","last_undegraded":"2024-09-13T07:25:49.890916+0000","last_fullsized":"2024-09-13T07:25:49.890916+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:42.260372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'97","reported_seq":141,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-13T07:25:48.883166+0000","last_change":"2024-09-13T07:25:42.741998+0000","last_active":"2024-09-13T07:25:48.883166+0000","last_peered":"2024-09-13T07:25:48.883166+0000","last_clean":"2024-09-13T07:25:48.883166+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:25:48.883166+0000","last_undegraded":"2024-09-13T07:25:48.883166+0000","last_fullsized":"2024-09-13T07:25:48.883166+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_clean_scrub_stamp":"2024-09-13T07:25:40.845594+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:10.726746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":208896,"data_stored":632997,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31260,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340580,"statfs":{"total":96636764160,"available":96604753920,"internally_reserved":0,"allocated":139264,"data_stored":39048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":208896,"data_stored":632997,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:25:54.781 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T07:25:54.781 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T07:25:54.781 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T07:25:54.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T07:25:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:25:55.119 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T07:25:55.133 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T07:25:55.133 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T07:25:55.143 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T07:25:55.143 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:25:55.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2024-09-13T07:25:55.606 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'datapool' created 2024-09-13T07:25:55.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2024-09-13T07:25:58.625 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T07:25:58.639 INFO:tasks.workunit:Pulling workunits from ref 7a853b1706d076d58e0da7a5d3b9b9eb36276880 2024-09-13T07:25:58.639 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T07:25:58.640 DEBUG:teuthology.orchestra.run.smithi005:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T07:25:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T07:25:58.646 INFO:teuthology.orchestra.run.smithi005.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T07:25:58.647 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T07:25:58.653 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T07:25:58.653 DEBUG:teuthology.orchestra.run.smithi005:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T07:25:58.701 INFO:tasks.workunit:timeout=3h 2024-09-13T07:25:58.701 INFO:tasks.workunit:cleanup=True 2024-09-13T07:25:58.702 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7a853b1706d076d58e0da7a5d3b9b9eb36276880 2024-09-13T07:25:58.711 INFO:tasks.workunit.client.0.smithi005.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T07:26:57.445 INFO:tasks.workunit.client.0.smithi005.stderr:Note: switching to '7a853b1706d076d58e0da7a5d3b9b9eb36276880'. 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr: git switch -c 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr:Or undo this operation with: 2024-09-13T07:26:57.446 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.447 INFO:tasks.workunit.client.0.smithi005.stderr: git switch - 2024-09-13T07:26:57.447 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.447 INFO:tasks.workunit.client.0.smithi005.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T07:26:57.447 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:26:57.447 INFO:tasks.workunit.client.0.smithi005.stderr:HEAD is now at 7a853b1706d Merge pull request #59683 from zdover23/wip-doc-2024-09-10-backport-59677-to-reef 2024-09-13T07:26:57.451 DEBUG:teuthology.orchestra.run.smithi005:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T07:26:57.482 INFO:tasks.workunit.client.0.smithi005.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T07:26:57.486 INFO:tasks.workunit.client.0.smithi005.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T07:26:57.486 INFO:tasks.workunit.client.0.smithi005.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T07:26:58.337 INFO:tasks.workunit.client.0.smithi005.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T07:26:58.431 INFO:tasks.workunit.client.0.smithi005.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T07:26:58.497 INFO:tasks.workunit.client.0.smithi005.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T07:26:58.500 INFO:tasks.workunit.client.0.smithi005.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T07:26:58.500 INFO:tasks.workunit.client.0.smithi005.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T07:26:58.560 INFO:tasks.workunit.client.0.smithi005.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T07:26:58.567 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T07:26:58.567 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T07:26:58.618 INFO:tasks.workunit:Running workunits matching rbd/cli_generic.sh on client.0... 2024-09-13T07:26:58.620 INFO:tasks.workunit:Running workunit rbd/cli_generic.sh... 2024-09-13T07:26:58.620 DEBUG:teuthology.orchestra.run.smithi005:workunit test rbd/cli_generic.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a853b1706d076d58e0da7a5d3b9b9eb36276880 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2024-09-13T07:26:58.635 INFO:tasks.workunit.client.0.smithi005.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2024-09-13T07:26:58.637 INFO:tasks.workunit.client.0.smithi005.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ TIMEOUT=300 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ MAX_TIMEOUT=15 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ PG_NUM=4 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ TMPDIR=/tmp 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ type xmlstarlet 2024-09-13T07:26:58.638 INFO:tasks.workunit.client.0.smithi005.stderr:++ XMLSTARLET=xmlstarlet 2024-09-13T07:26:58.639 INFO:tasks.workunit.client.0.smithi005.stderr:+++ uname 2024-09-13T07:26:58.639 INFO:tasks.workunit.client.0.smithi005.stderr:++ '[' Linux = FreeBSD ']' 2024-09-13T07:26:58.640 INFO:tasks.workunit.client.0.smithi005.stderr:++ SED=sed 2024-09-13T07:26:58.640 INFO:tasks.workunit.client.0.smithi005.stderr:++ AWK=awk 2024-09-13T07:26:58.641 INFO:tasks.workunit.client.0.smithi005.stderr:+++ stty -a 2024-09-13T07:26:58.641 INFO:tasks.workunit.client.0.smithi005.stderr:+++ head -1 2024-09-13T07:26:58.641 INFO:tasks.workunit.client.0.smithi005.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-13T07:26:58.648 INFO:tasks.workunit.client.0.smithi005.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T07:26:58.649 INFO:tasks.workunit.client.0.smithi005.stderr:++ termwidth= 2024-09-13T07:26:58.649 INFO:tasks.workunit.client.0.smithi005.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-13T07:26:58.649 INFO:tasks.workunit.client.0.smithi005.stderr:++ DIFFCOLOPTS='-y ' 2024-09-13T07:26:58.649 INFO:tasks.workunit.client.0.smithi005.stderr:++ KERNCORE=kernel.core_pattern 2024-09-13T07:26:58.650 INFO:tasks.workunit.client.0.smithi005.stderr:++ EXTRA_OPTS= 2024-09-13T07:26:58.658 INFO:tasks.workunit.client.0.smithi005.stderr:++ test '' = TESTS 2024-09-13T07:26:58.659 INFO:tasks.workunit.client.0.smithi005.stderr:+ export RBD_FORCE_ALLOW_V1=1 2024-09-13T07:26:58.659 INFO:tasks.workunit.client.0.smithi005.stderr:+ RBD_FORCE_ALLOW_V1=1 2024-09-13T07:26:58.660 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:26:58.660 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:26:58.660 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v '^0$' 2024-09-13T07:26:58.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ IMGS='testimg1 testimg2 testimg3 testimg4 testimg5 testimg6 testimg-diff1 testimg-diff2 testimg-diff3 foo foo2 bar bar2 test1 test2 test3 test4 clone2' 2024-09-13T07:26:58.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ tiered=0 2024-09-13T07:26:58.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd dump 2024-09-13T07:26:58.710 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^pool' 2024-09-13T07:26:58.710 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ''\''rbd'\''' 2024-09-13T07:26:58.710 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep tier 2024-09-13T07:26:59.127 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_pool_image_args 2024-09-13T07:26:59.128 INFO:tasks.workunit.client.0.smithi005.stdout:testing pool and image args... 2024-09-13T07:26:59.128 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing pool and image args...' 2024-09-13T07:26:59.128 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:26:59.128 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:26:59.243 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:26:59.355 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:26:59.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:26:59.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:26:59.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:26:59.879 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.144 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.535 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.662 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.788 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:00.918 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:01.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:01.171 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:01.296 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:01.418 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-13T07:27:01.837 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'test' does not exist 2024-09-13T07:27:01.846 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create test 32 2024-09-13T07:27:02.373 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'test' already exists 2024-09-13T07:27:02.382 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init test 2024-09-13T07:27:05.291 INFO:tasks.workunit.client.0.smithi005.stderr:+ truncate -s 1 /tmp/empty /tmp/empty@snap 2024-09-13T07:27:05.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:05.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:27:05.299 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:05.351 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:27:05.352 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 1 test1 2024-09-13T07:27:05.427 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:05.427 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test1 2024-09-13T07:27:05.472 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import --image test2 /tmp/empty 2024-09-13T07:27:05.559 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:05.560 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: --image is deprecated, use --dest 2024-09-13T07:27:05.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:05.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test2 2024-09-13T07:27:05.616 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --dest test3 import /tmp/empty 2024-09-13T07:27:05.710 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:05.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:05.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test3 2024-09-13T07:27:05.766 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import /tmp/empty foo 2024-09-13T07:27:06.063 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:06.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:06.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q foo 2024-09-13T07:27:06.118 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import --dest test/empty@snap /tmp/empty 2024-09-13T07:27:06.150 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-09-13T07:27:06.152 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:27:06.152 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import /tmp/empty test/empty@snap 2024-09-13T07:27:06.186 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-09-13T07:27:06.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:27:06.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import --image test/empty@snap /tmp/empty 2024-09-13T07:27:06.223 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-09-13T07:27:06.223 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: --image is deprecated, use --dest 2024-09-13T07:27:06.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:27:06.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import /tmp/empty@snap 2024-09-13T07:27:06.257 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2024-09-13T07:27:06.258 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:27:06.259 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:06.259 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:06.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:27:06.310 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:27:06.310 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import /tmp/empty test/test1 2024-09-13T07:27:06.399 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:06.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:06.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test1 2024-09-13T07:27:06.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p test import /tmp/empty test2 2024-09-13T07:27:06.552 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:06.553 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-09-13T07:27:06.557 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:06.557 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test2 2024-09-13T07:27:06.604 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --image test3 -p test import /tmp/empty 2024-09-13T07:27:06.695 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:06.695 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: --image is deprecated, use --dest 2024-09-13T07:27:06.696 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-09-13T07:27:06.701 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:06.701 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test3 2024-09-13T07:27:06.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --image test4 -p test import /tmp/empty 2024-09-13T07:27:06.847 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:06.848 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: --image is deprecated, use --dest 2024-09-13T07:27:06.848 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-09-13T07:27:06.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:06.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test4 2024-09-13T07:27:06.903 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --dest test5 -p test import /tmp/empty 2024-09-13T07:27:06.991 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:06.992 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-09-13T07:27:06.997 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:06.997 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test5 2024-09-13T07:27:07.050 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --dest test6 --dest-pool test import /tmp/empty 2024-09-13T07:27:07.145 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:07.151 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:07.151 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test6 2024-09-13T07:27:07.202 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --image test7 --dest-pool test import /tmp/empty 2024-09-13T07:27:07.295 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:07.295 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: --image is deprecated, use --dest 2024-09-13T07:27:07.301 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:07.301 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test7 2024-09-13T07:27:07.354 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --image test/test8 import /tmp/empty 2024-09-13T07:27:07.448 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:07.449 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: --image is deprecated, use --dest 2024-09-13T07:27:07.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:07.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test8 2024-09-13T07:27:07.503 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --dest test/test9 import /tmp/empty 2024-09-13T07:27:07.597 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:07.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:07.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test9 2024-09-13T07:27:07.656 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import --pool test /tmp/empty 2024-09-13T07:27:07.751 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:27:07.752 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2024-09-13T07:27:07.757 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:07.758 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q empty 2024-09-13T07:27:07.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy test/test9 test10 2024-09-13T07:27:07.914 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-13T07:27:07.922 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:07.922 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -qv test10 2024-09-13T07:27:07.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:07.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test10 2024-09-13T07:27:08.025 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy test/test9 test/test10 2024-09-13T07:27:08.129 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-13T07:27:08.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:08.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test10 2024-09-13T07:27:08.186 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy --pool test test10 --dest-pool test test11 2024-09-13T07:27:08.290 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-13T07:27:08.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:08.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -q test11 2024-09-13T07:27:08.343 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy --dest-pool rbd --pool test test11 test12 2024-09-13T07:27:08.452 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-13T07:27:08.458 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:08.458 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test12 2024-09-13T07:27:08.511 INFO:tasks.workunit.client.0.smithi005.stdout:test12 2024-09-13T07:27:08.511 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls test 2024-09-13T07:27:08.512 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -qv test12 2024-09-13T07:27:08.562 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -f /tmp/empty /tmp/empty@snap 2024-09-13T07:27:08.563 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-13T07:27:09.401 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'test' does not exist 2024-09-13T07:27:09.410 INFO:tasks.workunit.client.0.smithi005.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-09-13T07:27:09.410 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm foo 2024-09-13T07:27:09.543 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:09.541+0000 7f8da8d1c700 0 -- 172.21.15.5:0/3300089936 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f8d8c039cc0 msgr2=0x7f8d8c03c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:09.569 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:09.569+0000 7f8da8d1c700 0 -- 172.21.15.5:0/3300089936 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f8d84016890 msgr2=0x7f8d8c03ecd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:09.576 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:09.582 INFO:tasks.workunit.client.0.smithi005.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-09-13T07:27:09.582 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:27:09.740 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:09.737+0000 7fa726bc9700 0 -- 172.21.15.5:0/1985411239 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x565059b360d0 msgr2=0x565059b4e2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:09.756 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:09.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-09-13T07:27:09.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test10 2024-09-13T07:27:09.917 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:09.913+0000 7f4206775700 0 -- 172.21.15.5:0/2886713181 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5654a3969540 msgr2=0x5654a396b9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:09.934 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:09.939 INFO:tasks.workunit.client.0.smithi005.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-09-13T07:27:09.939 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test12 2024-09-13T07:27:10.082 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:10.081+0000 7f61892b3700 0 -- 172.21.15.5:0/2976939659 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f6168008be0 msgr2=0x7f616800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:10.105 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:10.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-09-13T07:27:10.110 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:27:10.261 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:10.261+0000 7ffad253d700 0 -- 172.21.15.5:0/4114090615 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cc093751c0 msgr2=0x55cc093883b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:10.280 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:10.285 INFO:tasks.workunit.client.0.smithi005.stderr:+ for f in foo test1 test10 test12 test2 test3 2024-09-13T07:27:10.285 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test3 2024-09-13T07:27:10.434 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:10.433+0000 7ff43934d700 0 -- 172.21.15.5:0/3791294315 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff414004640 msgr2=0x7ff4140565d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:10.454 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:10.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_rename 2024-09-13T07:27:10.460 INFO:tasks.workunit.client.0.smithi005.stdout:testing rename... 2024-09-13T07:27:10.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing rename...' 2024-09-13T07:27:10.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:27:10.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:10.584 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:10.707 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:10.842 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:10.972 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.232 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.361 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.627 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:11.885 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.141 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.266 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.394 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.519 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.855 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:12.986 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 1 foo 2024-09-13T07:27:13.022 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:27:13.035 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:13.033+0000 7f243e2c0780 -1 librbd: Forced V1 image creation. 2024-09-13T07:27:13.046 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 bar 2024-09-13T07:27:13.130 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename foo foo2 2024-09-13T07:27:13.210 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename foo2 bar 2024-09-13T07:27:13.210 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep exists 2024-09-13T07:27:13.274 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13T07:27:13.265+0000 7f1ac76a5780 -1 librbd::Operations: rbd image bar already exists 2024-09-13T07:27:13.274 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: rename error: (17) File exists 2024-09-13T07:27:13.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename bar bar2 2024-09-13T07:27:13.384 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename bar2 foo2 2024-09-13T07:27:13.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep exists 2024-09-13T07:27:13.456 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13T07:27:13.445+0000 7f27b9315780 -1 librbd::Operations: rbd image foo2 already exists 2024-09-13T07:27:13.456 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: rename error: (17) File exists 2024-09-13T07:27:13.456 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:27:14.419 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:27:14.432 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:27:17.341 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -p rbd2 -s 1 foo 2024-09-13T07:27:17.417 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename rbd2/foo rbd2/bar 2024-09-13T07:27:17.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p rbd2 ls 2024-09-13T07:27:17.529 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep bar 2024-09-13T07:27:17.579 INFO:tasks.workunit.client.0.smithi005.stdout:bar 2024-09-13T07:27:17.580 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename rbd2/bar foo 2024-09-13T07:27:17.691 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename --pool rbd2 foo bar 2024-09-13T07:27:17.937 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:17.936+0000 7f421735b700 0 --2- 172.21.15.5:0/3214723771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5555c6e35980 0x5555c6e258b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:27:18.010 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename rbd2/bar --dest-pool rbd foo 2024-09-13T07:27:18.045 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: mv/rename across pools not supported 2024-09-13T07:27:18.045 INFO:tasks.workunit.client.0.smithi005.stderr:source pool: rbd2 dest pool: rbd 2024-09-13T07:27:18.047 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rename --pool rbd2 bar --dest-pool rbd2 foo 2024-09-13T07:27:18.159 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p rbd2 ls 2024-09-13T07:27:18.159 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep foo 2024-09-13T07:27:18.212 INFO:tasks.workunit.client.0.smithi005.stdout:foo 2024-09-13T07:27:18.213 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:27:19.451 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:27:19.465 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:27:19.465 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:19.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:19.708 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:19.839 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:19.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.103 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.231 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.369 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.503 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.638 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:20.912 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:21.246 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:21.497 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:21.621 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:21.747 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:21.876 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.007 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.138 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_ls 2024-09-13T07:27:22.138 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing ls...' 2024-09-13T07:27:22.138 INFO:tasks.workunit.client.0.smithi005.stdout:testing ls... 2024-09-13T07:27:22.138 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:27:22.139 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.268 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.397 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.523 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.786 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:22.916 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.049 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.309 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.435 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.702 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.825 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:23.955 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:24.086 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:24.217 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:24.348 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:24.478 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 1 test1 2024-09-13T07:27:24.514 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:27:24.528 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:24.524+0000 7f56e07be780 -1 librbd: Forced V1 image creation. 2024-09-13T07:27:24.538 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 1 test2 2024-09-13T07:27:24.574 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:27:24.587 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:24.584+0000 7fb000759780 -1 librbd: Forced V1 image creation. 2024-09-13T07:27:24.600 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:24.600 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:27:24.649 INFO:tasks.workunit.client.0.smithi005.stdout:test1 2024-09-13T07:27:24.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:24.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:27:24.698 INFO:tasks.workunit.client.0.smithi005.stdout:test2 2024-09-13T07:27:24.699 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:24.699 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:24.699 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:27:24.748 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:27:24.749 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:24.749 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*1 MiB.*1' 2024-09-13T07:27:24.808 INFO:tasks.workunit.client.0.smithi005.stdout:test1 1 MiB 1 2024-09-13T07:27:24.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:24.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*1 MiB.*1' 2024-09-13T07:27:24.874 INFO:tasks.workunit.client.0.smithi005.stdout:test2 1 MiB 1 2024-09-13T07:27:24.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:27:24.940 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:24.945 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:27:25.014 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:25.019 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:27:25.098 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:27:25.181 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:25.181 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:27:25.230 INFO:tasks.workunit.client.0.smithi005.stdout:test1 2024-09-13T07:27:25.231 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:25.231 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:27:25.282 INFO:tasks.workunit.client.0.smithi005.stdout:test2 2024-09-13T07:27:25.283 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:25.283 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:25.283 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:27:25.332 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:27:25.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:25.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*1 MiB.*2' 2024-09-13T07:27:25.400 INFO:tasks.workunit.client.0.smithi005.stdout:test1 1 MiB 2 2024-09-13T07:27:25.401 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:25.401 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*1 MiB.*2' 2024-09-13T07:27:25.468 INFO:tasks.workunit.client.0.smithi005.stdout:test2 1 MiB 2 2024-09-13T07:27:25.468 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:27:25.615 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:25.612+0000 7eff1d402700 0 -- 172.21.15.5:0/3172099852 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563eacb22100 msgr2=0x563eacb3a350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:25.633 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:25.638 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:27:25.789 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:25.784+0000 7f7599dfa700 0 -- 172.21.15.5:0/1436411583 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5566dc74e100 msgr2=0x5566dc766350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:25.807 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:25.812 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:27:25.898 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 1 test2 2024-09-13T07:27:25.934 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:27:25.947 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:25.944+0000 7f0ee3b96780 -1 librbd: Forced V1 image creation. 2024-09-13T07:27:25.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:25.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:27:26.006 INFO:tasks.workunit.client.0.smithi005.stdout:test1 2024-09-13T07:27:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:26.007 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:27:26.055 INFO:tasks.workunit.client.0.smithi005.stdout:test2 2024-09-13T07:27:26.056 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:26.056 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:27:26.056 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:26.104 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:27:26.105 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:26.105 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*1 MiB.*2' 2024-09-13T07:27:26.164 INFO:tasks.workunit.client.0.smithi005.stdout:test1 1 MiB 2 2024-09-13T07:27:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:26.165 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*1 MiB.*1' 2024-09-13T07:27:26.234 INFO:tasks.workunit.client.0.smithi005.stdout:test2 1 MiB 1 2024-09-13T07:27:26.234 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:27:26.234 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:26.359 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:26.493 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:26.624 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:26.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:26.882 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.145 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.544 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.674 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:27.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:28.186 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:28.326 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:28.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:28.580 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:27:28.712 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq -w 00 99 2024-09-13T07:27:28.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:28.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.00 -s 1 2024-09-13T07:27:28.966 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:28.964+0000 7f105f6c0700 0 --2- 172.21.15.5:0/637126123 >> v2:172.21.15.5:3300/0 conn(0x55a80fcda9a0 0x55a80fcdcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:27:29.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.01 -s 1 2024-09-13T07:27:29.096 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.096 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.02 -s 1 2024-09-13T07:27:29.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.03 -s 1 2024-09-13T07:27:29.252 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.252 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.04 -s 1 2024-09-13T07:27:29.338 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.338 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.05 -s 1 2024-09-13T07:27:29.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.06 -s 1 2024-09-13T07:27:29.485 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.485 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.07 -s 1 2024-09-13T07:27:29.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.08 -s 1 2024-09-13T07:27:29.650 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.650 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.09 -s 1 2024-09-13T07:27:29.729 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.729 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.10 -s 1 2024-09-13T07:27:29.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.11 -s 1 2024-09-13T07:27:29.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.12 -s 1 2024-09-13T07:27:29.962 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:29.962 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.13 -s 1 2024-09-13T07:27:30.047 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.047 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.14 -s 1 2024-09-13T07:27:30.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.15 -s 1 2024-09-13T07:27:30.208 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.16 -s 1 2024-09-13T07:27:30.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.17 -s 1 2024-09-13T07:27:30.374 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.374 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.18 -s 1 2024-09-13T07:27:30.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.19 -s 1 2024-09-13T07:27:30.535 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.536 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.20 -s 1 2024-09-13T07:27:30.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.21 -s 1 2024-09-13T07:27:30.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.22 -s 1 2024-09-13T07:27:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.774 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.23 -s 1 2024-09-13T07:27:30.859 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.859 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.24 -s 1 2024-09-13T07:27:30.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:30.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.25 -s 1 2024-09-13T07:27:31.015 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.016 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.26 -s 1 2024-09-13T07:27:31.094 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.094 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.27 -s 1 2024-09-13T07:27:31.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.28 -s 1 2024-09-13T07:27:31.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.29 -s 1 2024-09-13T07:27:31.337 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.337 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.30 -s 1 2024-09-13T07:27:31.412 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.31 -s 1 2024-09-13T07:27:31.490 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.491 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.32 -s 1 2024-09-13T07:27:31.574 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.574 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.33 -s 1 2024-09-13T07:27:31.657 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.657 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.34 -s 1 2024-09-13T07:27:31.738 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.738 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.35 -s 1 2024-09-13T07:27:31.815 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.815 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.36 -s 1 2024-09-13T07:27:31.891 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.891 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.37 -s 1 2024-09-13T07:27:31.972 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:31.973 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.38 -s 1 2024-09-13T07:27:32.048 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.048 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.39 -s 1 2024-09-13T07:27:32.129 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.129 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.40 -s 1 2024-09-13T07:27:32.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.41 -s 1 2024-09-13T07:27:32.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.42 -s 1 2024-09-13T07:27:32.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.43 -s 1 2024-09-13T07:27:32.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.44 -s 1 2024-09-13T07:27:32.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.45 -s 1 2024-09-13T07:27:32.616 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.616 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.46 -s 1 2024-09-13T07:27:32.690 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.690 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.47 -s 1 2024-09-13T07:27:32.774 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.774 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.48 -s 1 2024-09-13T07:27:32.853 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.853 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.49 -s 1 2024-09-13T07:27:32.939 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:32.939 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.50 -s 1 2024-09-13T07:27:33.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.51 -s 1 2024-09-13T07:27:33.102 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.102 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.52 -s 1 2024-09-13T07:27:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.53 -s 1 2024-09-13T07:27:33.266 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.266 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.54 -s 1 2024-09-13T07:27:33.347 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.347 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.55 -s 1 2024-09-13T07:27:33.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.56 -s 1 2024-09-13T07:27:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.504 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.57 -s 1 2024-09-13T07:27:33.584 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.58 -s 1 2024-09-13T07:27:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.664 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.59 -s 1 2024-09-13T07:27:33.745 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.745 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.60 -s 1 2024-09-13T07:27:33.829 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.829 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.61 -s 1 2024-09-13T07:27:33.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.62 -s 1 2024-09-13T07:27:33.992 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:33.993 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.63 -s 1 2024-09-13T07:27:34.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.64 -s 1 2024-09-13T07:27:34.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.65 -s 1 2024-09-13T07:27:34.230 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.230 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.66 -s 1 2024-09-13T07:27:34.308 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.309 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.67 -s 1 2024-09-13T07:27:34.390 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.390 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.68 -s 1 2024-09-13T07:27:34.470 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.471 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.69 -s 1 2024-09-13T07:27:34.554 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.554 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.70 -s 1 2024-09-13T07:27:34.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.71 -s 1 2024-09-13T07:27:34.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.72 -s 1 2024-09-13T07:27:34.799 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.800 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.73 -s 1 2024-09-13T07:27:34.880 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.880 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.74 -s 1 2024-09-13T07:27:34.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:34.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.75 -s 1 2024-09-13T07:27:35.044 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.044 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.76 -s 1 2024-09-13T07:27:35.124 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.77 -s 1 2024-09-13T07:27:35.205 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.206 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.78 -s 1 2024-09-13T07:27:35.288 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.288 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.79 -s 1 2024-09-13T07:27:35.370 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.371 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.80 -s 1 2024-09-13T07:27:35.448 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.448 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.81 -s 1 2024-09-13T07:27:35.531 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.531 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.82 -s 1 2024-09-13T07:27:35.613 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.613 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.83 -s 1 2024-09-13T07:27:35.693 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.693 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.84 -s 1 2024-09-13T07:27:35.775 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.775 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.85 -s 1 2024-09-13T07:27:35.858 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.858 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.86 -s 1 2024-09-13T07:27:35.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:35.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.87 -s 1 2024-09-13T07:27:36.015 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.015 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.88 -s 1 2024-09-13T07:27:36.099 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.099 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.89 -s 1 2024-09-13T07:27:36.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.90 -s 1 2024-09-13T07:27:36.258 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.258 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.91 -s 1 2024-09-13T07:27:36.548 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.548 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.92 -s 1 2024-09-13T07:27:36.626 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.626 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.93 -s 1 2024-09-13T07:27:36.703 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.703 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.94 -s 1 2024-09-13T07:27:36.784 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.785 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.95 -s 1 2024-09-13T07:27:36.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:36.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.96 -s 1 2024-09-13T07:27:37.109 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:37.107+0000 7f55e55c3700 0 --2- 172.21.15.5:0/760046065 >> v2:172.21.15.5:3300/0 conn(0x5588d24c27a0 0x5588d24c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:27:37.159 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:37.159 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.97 -s 1 2024-09-13T07:27:37.239 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:37.239 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.98 -s 1 2024-09-13T07:27:37.324 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:37.324 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.99 -s 1 2024-09-13T07:27:37.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:27:37.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:37.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 100 2024-09-13T07:27:37.458 INFO:tasks.workunit.client.0.smithi005.stdout:100 2024-09-13T07:27:37.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:27:37.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep image 2024-09-13T07:27:37.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:27:37.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 100 2024-09-13T07:27:37.541 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:37.535+0000 7f69ce6e5700 0 -- 172.21.15.5:0/2912781325 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cb9b74be40 msgr2=0x55cb9b75f360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:37.545 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:37.539+0000 7f69ce6e5700 0 -- 172.21.15.5:0/2912781325 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cb9b74be40 msgr2=0x7f69b003ed90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:37.821 INFO:tasks.workunit.client.0.smithi005.stdout:100 2024-09-13T07:27:37.821 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq -w 00 99 2024-09-13T07:27:37.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:37.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.00 2024-09-13T07:27:37.967 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:37.963+0000 7f32cc429700 0 -- 172.21.15.5:0/2893796591 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f32a8008be0 msgr2=0x7f32a800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:37.986 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:37.991 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:37.991 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.01 2024-09-13T07:27:38.142 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:38.139+0000 7f1706f5d700 0 -- 172.21.15.5:0/2693653991 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f16e8006810 msgr2=0x7f16e8009560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:38.162 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:38.167 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:38.167 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.02 2024-09-13T07:27:38.325 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:38.319+0000 7ff4506d4700 0 -- 172.21.15.5:0/679471187 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff434008d50 msgr2=0x7ff43400b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:38.346 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:38.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:38.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.03 2024-09-13T07:27:38.503 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:38.499+0000 7f1ec4a4e700 0 -- 172.21.15.5:0/3135063946 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5609232200d0 msgr2=0x5609232382c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:38.519 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:38.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:38.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.04 2024-09-13T07:27:38.681 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:38.675+0000 7f97d080c700 0 -- 172.21.15.5:0/3498928636 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f97ac008be0 msgr2=0x7f97ac00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:38.702 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:38.707 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:38.707 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.05 2024-09-13T07:27:38.869 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:38.863+0000 7f297d57a700 0 -- 172.21.15.5:0/2491710841 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d3d6f6a0d0 msgr2=0x55d3d6f822c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:38.885 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:38.890 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:38.890 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.06 2024-09-13T07:27:39.049 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.043+0000 7f65de256700 0 -- 172.21.15.5:0/1201665635 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f1c519a100 msgr2=0x55f1c51b2800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.065 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:39.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:39.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.07 2024-09-13T07:27:39.223 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.219+0000 7f43d25d7700 0 -- 172.21.15.5:0/526928542 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f43b0008be0 msgr2=0x7f43b000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.247 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.243+0000 7f43d403a700 0 -- 172.21.15.5:0/526928542 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f43b8039c70 msgr2=0x7f43b803c080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.249 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:39.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:39.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.08 2024-09-13T07:27:39.410 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.403+0000 7fa8731a3700 0 -- 172.21.15.5:0/1666943528 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55bcc4ae3770 msgr2=0x55bcc4ae5c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.426 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:39.432 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:39.432 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.09 2024-09-13T07:27:39.588 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.583+0000 7fac8b0f8700 0 -- 172.21.15.5:0/2453835724 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x558c37d14350 msgr2=0x558c37d167e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.606 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:39.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:39.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.10 2024-09-13T07:27:39.765 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.759+0000 7fbe1b853700 0 -- 172.21.15.5:0/3639840349 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fbdfc006b90 msgr2=0x7fbdfc006f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.788 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:39.792 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:39.792 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.11 2024-09-13T07:27:39.946 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:39.943+0000 7ff139196700 0 -- 172.21.15.5:0/3081963650 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x556563eea0d0 msgr2=0x556563f022c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:39.963 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:39.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:39.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.12 2024-09-13T07:27:40.121 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:40.115+0000 7f7d3d69a700 0 -- 172.21.15.5:0/3489876851 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e01f4e5100 msgr2=0x55e01f4fd350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:40.141 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:40.146 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:40.146 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.13 2024-09-13T07:27:40.299 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:40.295+0000 7f56acbe7700 0 -- 172.21.15.5:0/4237559316 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ab53c700d0 msgr2=0x55ab53c882c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:40.322 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:40.327 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:40.327 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.14 2024-09-13T07:27:40.479 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:40.475+0000 7f38e9f50700 0 -- 172.21.15.5:0/3516380783 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b07f0c0100 msgr2=0x55b07f0d8800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:40.498 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:40.503 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:40.503 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.15 2024-09-13T07:27:40.661 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:40.655+0000 7fe496f76700 0 -- 172.21.15.5:0/1286825632 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x562e8adf71e0 msgr2=0x562e8adf9670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:40.678 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:40.683 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:40.683 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.16 2024-09-13T07:27:40.835 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:40.831+0000 7efe43076700 0 -- 172.21.15.5:0/2553847885 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7efe20008be0 msgr2=0x7efe2000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:40.858 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:40.855+0000 7efe44ad9700 0 -- 172.21.15.5:0/2553847885 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7efe28039cc0 msgr2=0x7efe2803c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:40.864 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:40.869 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:40.869 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.17 2024-09-13T07:27:41.030 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:41.023+0000 7f958ab69700 0 -- 172.21.15.5:0/3878948438 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f956c058310 msgr2=0x7f956c006140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:41.051 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:41.057 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:41.057 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.18 2024-09-13T07:27:41.212 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:41.207+0000 7f892df6a700 0 -- 172.21.15.5:0/768899923 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x559a17a7c1c0 msgr2=0x559a17a8f320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:41.228 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:41.233 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:41.233 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.19 2024-09-13T07:27:41.396 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:41.391+0000 7f740ffff700 0 -- 172.21.15.5:0/1168840598 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f73f0056680 msgr2=0x7f73f0055bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:41.417 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:41.421 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:41.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.20 2024-09-13T07:27:41.780 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:41.775+0000 7f439837b700 0 -- 172.21.15.5:0/2498241490 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5600d8e0b010 msgr2=0x5600d8e0d4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:41.804 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:41.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:41.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.21 2024-09-13T07:27:41.965 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:41.959+0000 7fc2cb051700 0 -- 172.21.15.5:0/4181280019 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5589c04cdc00 msgr2=0x7fc2ac03ed50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:41.984 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:41.979+0000 7fc2cb051700 0 -- 172.21.15.5:0/4181280019 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fc2ac039cc0 msgr2=0x7fc2ac03c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:41.995 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:42.000 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:42.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.22 2024-09-13T07:27:42.157 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:42.151+0000 7f798883e700 0 -- 172.21.15.5:0/2115536556 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55fa21f36290 msgr2=0x55fa21f38720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:42.177 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:42.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:42.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.23 2024-09-13T07:27:42.346 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:42.339+0000 7fa947a13700 0 -- 172.21.15.5:0/1925671914 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fa92005be20 msgr2=0x7fa920068b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:42.369 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:42.373 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:42.373 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.24 2024-09-13T07:27:42.528 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:42.523+0000 7f293b5b1700 0 -- 172.21.15.5:0/1807826973 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55675301b3a0 msgr2=0x55675301d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:42.548 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:42.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:42.554 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.25 2024-09-13T07:27:42.717 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:42.711+0000 7f7f0243d700 0 -- 172.21.15.5:0/1899357090 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f7ee8008be0 msgr2=0x7f7ee800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:42.740 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:42.745 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:42.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.26 2024-09-13T07:27:42.909 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:42.903+0000 7f1a86b30700 0 -- 172.21.15.5:0/1482399889 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e7f3c80f80 msgr2=0x55e7f3c99150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:42.932 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:42.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:42.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.27 2024-09-13T07:27:43.086 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:43.079+0000 7f1ea7b6d700 0 -- 172.21.15.5:0/3943690626 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1e88039c60 msgr2=0x7f1e8803c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:43.108 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:43.103+0000 7f1ea95d0700 0 -- 172.21.15.5:0/3943690626 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e3c5fbc0d0 msgr2=0x55e3c5fd42c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:43.115 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:43.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:43.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.28 2024-09-13T07:27:43.292 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:43.287+0000 7f14948ed700 0 -- 172.21.15.5:0/4143281227 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1470008be0 msgr2=0x7f147000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:43.313 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:43.318 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:43.318 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.29 2024-09-13T07:27:43.480 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:43.475+0000 7fbcb428c700 0 -- 172.21.15.5:0/3546984669 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5563a19661c0 msgr2=0x5563a19793b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:43.499 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:43.504 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:43.505 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.30 2024-09-13T07:27:43.662 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:43.659+0000 7f22c917d700 0 -- 172.21.15.5:0/1106216657 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d47dc203a0 msgr2=0x55d47dc38350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:43.686 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:43.691 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:43.692 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.31 2024-09-13T07:27:43.856 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:43.851+0000 7f0cfa93e700 0 -- 172.21.15.5:0/1319162602 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56535f3250d0 msgr2=0x56535f33d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:43.878 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:43.884 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:43.884 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.32 2024-09-13T07:27:44.046 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:44.043+0000 7ff44cb5e700 0 -- 172.21.15.5:0/3424814829 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55af1508a3a0 msgr2=0x55af1508c790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:44.064 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:44.068 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:44.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.33 2024-09-13T07:27:44.226 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:44.223+0000 7eff94576700 0 -- 172.21.15.5:0/125389987 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56471fedb1c0 msgr2=0x56471feee320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:44.250 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:44.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:44.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.34 2024-09-13T07:27:44.418 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:44.415+0000 7f50a260c700 0 -- 172.21.15.5:0/462929640 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x559e675010d0 msgr2=0x559e675192c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:44.437 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:44.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:44.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.35 2024-09-13T07:27:44.597 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:44.591+0000 7efce1e2f700 0 -- 172.21.15.5:0/2110256130 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f41660b100 msgr2=0x55f416623310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:44.618 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:44.624 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:44.624 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.36 2024-09-13T07:27:44.789 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:44.783+0000 7fd010222700 0 -- 172.21.15.5:0/1341307594 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564e23246100 msgr2=0x564e2325e350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:44.808 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:44.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.37 2024-09-13T07:27:44.972 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:44.967+0000 7fe76cff0700 0 -- 172.21.15.5:0/2652392148 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c47f393240 msgr2=0x55c47f3a7f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:44.994 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:44.999 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:44.999 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.38 2024-09-13T07:27:45.160 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:45.155+0000 7fbd9c59c700 0 -- 172.21.15.5:0/1582512899 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55b7f4b2d460 msgr2=0x55b7f4b2f8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:45.179 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:45.185 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:45.185 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.39 2024-09-13T07:27:45.350 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:45.347+0000 7fcf88b6d700 0 -- 172.21.15.5:0/2002069410 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5635aace8f80 msgr2=0x5635aad01150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:45.370 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:45.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:45.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.40 2024-09-13T07:27:45.533 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:45.527+0000 7fbb0aa4a700 0 -- 172.21.15.5:0/3189086254 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5610cd67c0d0 msgr2=0x5610cd6942c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:45.555 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:45.560 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:45.561 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.41 2024-09-13T07:27:45.719 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:45.715+0000 7f38cbb99700 0 -- 172.21.15.5:0/801306178 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f38b0039cc0 msgr2=0x7f38b003c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:45.742 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:45.739+0000 7f38cd5fc700 0 -- 172.21.15.5:0/801306178 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5606015ed9d0 msgr2=0x560601603150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:45.748 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:45.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:45.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.42 2024-09-13T07:27:45.915 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:45.911+0000 7f3aad373700 0 -- 172.21.15.5:0/2708152871 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563cb2895630 msgr2=0x563cb2897a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:45.936 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:45.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:45.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.43 2024-09-13T07:27:46.099 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:46.095+0000 7f8ec1a47700 0 -- 172.21.15.5:0/2552995705 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f8e9c006f20 msgr2=0x7f8e9c05b7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:46.120 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:46.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:46.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.44 2024-09-13T07:27:46.284 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:46.279+0000 7ff43f6c3700 0 -- 172.21.15.5:0/978169984 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ff420039cc0 msgr2=0x7ff42003c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:46.307 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:46.312 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:46.312 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.45 2024-09-13T07:27:46.472 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:46.467+0000 7fcf178dd700 0 -- 172.21.15.5:0/81659960 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fcef4008be0 msgr2=0x7fcef400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:46.494 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:46.500 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:46.500 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.46 2024-09-13T07:27:46.668 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:46.663+0000 7f1a56101700 0 -- 172.21.15.5:0/2317124119 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55bf996cf1c0 msgr2=0x55bf996e2360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:46.688 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:46.683+0000 7f1a56101700 0 -- 172.21.15.5:0/2317124119 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f1a34039c60 msgr2=0x7f1a3403c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:46.688 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:46.692 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:46.693 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.47 2024-09-13T07:27:46.850 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:46.847+0000 7f9e570ef700 0 -- 172.21.15.5:0/1487069019 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ab63a31100 msgr2=0x55ab63a49310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:46.873 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:46.878 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:46.878 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.48 2024-09-13T07:27:47.029 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:47.023+0000 7fb9e4e17700 0 -- 172.21.15.5:0/2883342134 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fb9c0008be0 msgr2=0x7fb9c000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:47.053 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:47.047+0000 7fb9e4e17700 0 -- 172.21.15.5:0/2883342134 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x5593d6cefc00 msgr2=0x7fb9c803ed50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:47.059 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:47.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:47.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.49 2024-09-13T07:27:47.218 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:47.215+0000 7fd556b2e700 0 -- 172.21.15.5:0/1853978284 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5604ddba80d0 msgr2=0x5604ddbc02c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:47.240 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:47.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:47.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.50 2024-09-13T07:27:47.404 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:47.399+0000 7fd126313700 0 -- 172.21.15.5:0/570065934 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fd1080079d0 msgr2=0x7fd108009dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:47.631 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:47.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:47.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.51 2024-09-13T07:27:47.797 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:47.791+0000 7fceecac6700 0 -- 172.21.15.5:0/431721082 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56468231a100 msgr2=0x564682332350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:47.819 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:47.824 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:47.824 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.52 2024-09-13T07:27:47.988 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:47.983+0000 7fdd77ca8700 0 -- 172.21.15.5:0/2447135068 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b246da2c00 msgr2=0x55b246e610c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.007 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:48.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:48.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.53 2024-09-13T07:27:48.165 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.159+0000 7ff402447700 0 -- 172.21.15.5:0/3388765100 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ff3e4039cc0 msgr2=0x7ff3e403c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.191 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.187+0000 7ff402447700 0 -- 172.21.15.5:0/3388765100 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e15d0fe240 msgr2=0x7ff3e403ed70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:48.196 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:48.202 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:48.202 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.54 2024-09-13T07:27:48.361 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.355+0000 7f074f66a700 0 -- 172.21.15.5:0/4271229377 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5597424b9100 msgr2=0x5597424d1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.381 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:48.386 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:48.386 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.55 2024-09-13T07:27:48.554 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.547+0000 7f12cc373700 0 -- 172.21.15.5:0/442315422 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5593647ef0d0 msgr2=0x5593648072c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.573 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:48.577 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:48.578 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.56 2024-09-13T07:27:48.739 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.735+0000 7ff80dbcd700 0 -- 172.21.15.5:0/2810576306 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ff7f0039c70 msgr2=0x7ff7f003c080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.759 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.755+0000 7ff80dbcd700 0 -- 172.21.15.5:0/2810576306 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557e61cb3610 msgr2=0x7ff7f003f1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.760 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:48.765 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:48.765 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.57 2024-09-13T07:27:48.919 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.915+0000 7f1d755b4700 0 -- 172.21.15.5:0/3409246522 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f1d58039d80 msgr2=0x7f1d5803c190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.943 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:48.939+0000 7f1d76816700 0 -- 172.21.15.5:0/3409246522 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555a9a5651c0 msgr2=0x555a9a577f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:48.943 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:48.948 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:48.949 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.58 2024-09-13T07:27:49.110 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:49.107+0000 7f7875efa700 0 -- 172.21.15.5:0/4043060749 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555db49760d0 msgr2=0x555db498e2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:49.133 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:49.138 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:49.138 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.59 2024-09-13T07:27:49.294 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:49.287+0000 7efe82e40700 0 -- 172.21.15.5:0/1379414005 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7efe64016990 msgr2=0x7efe64016da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:49.315 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:49.320 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:49.320 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.60 2024-09-13T07:27:49.478 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:49.471+0000 7fe9a8388700 0 -- 172.21.15.5:0/4080078779 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c41a93d1c0 msgr2=0x55c41a94ff80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:49.502 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:49.507 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:49.508 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.61 2024-09-13T07:27:49.669 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:49.663+0000 7f9f784fe700 0 -- 172.21.15.5:0/1637176335 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f9f5c008be0 msgr2=0x7f9f5c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:49.691 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:49.696 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:49.696 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.62 2024-09-13T07:27:49.860 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:49.855+0000 7fe1a7f79700 0 -- 172.21.15.5:0/4144797699 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f8ae665c00 msgr2=0x55f8ae7242e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:49.878 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:49.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:49.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.63 2024-09-13T07:27:50.035 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.031+0000 7ff8a64a0700 0 -- 172.21.15.5:0/955878786 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560491b960d0 msgr2=0x560491babda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:50.057 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.051+0000 7ff8a64a0700 0 -- 172.21.15.5:0/955878786 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ff88803c830 msgr2=0x7ff88803ec40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:50.058 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:50.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:50.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.64 2024-09-13T07:27:50.198 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.195+0000 7f2987fff700 0 -- 172.21.15.5:0/3558111019 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f2968008be0 msgr2=0x7f296800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:50.222 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:50.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:50.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.65 2024-09-13T07:27:50.380 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.375+0000 7f1173fff700 0 -- 172.21.15.5:0/1028397069 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1154008be0 msgr2=0x7f115400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:50.406 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:50.411 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:50.411 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.66 2024-09-13T07:27:50.576 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.571+0000 7f00546e0700 0 -- 172.21.15.5:0/1261630165 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5555e62110d0 msgr2=0x5555e62292c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:50.594 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:50.599 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:50.599 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.67 2024-09-13T07:27:50.770 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.767+0000 7f1d55d23700 0 -- 172.21.15.5:0/2643739379 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1d3c008be0 msgr2=0x7f1d3c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:50.793 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:50.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:50.799 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.68 2024-09-13T07:27:50.959 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:50.955+0000 7fd4c557e700 0 -- 172.21.15.5:0/2471945056 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ea155be240 msgr2=0x55ea155d4440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:50.977 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:50.982 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:50.982 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.69 2024-09-13T07:27:51.150 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:51.143+0000 7f98609c2700 0 -- 172.21.15.5:0/1864690508 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a3c40569d0 msgr2=0x55a3c406c150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:51.169 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:51.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:51.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.70 2024-09-13T07:27:51.324 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:51.319+0000 7fe9f007f700 0 -- 172.21.15.5:0/3544602463 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fe9d4039cb0 msgr2=0x7fe9d403c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:51.347 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:51.343+0000 7fe9f007f700 0 -- 172.21.15.5:0/3544602463 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fe9cc008be0 msgr2=0x7fe9d403f760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:51.353 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:51.358 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:51.358 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.71 2024-09-13T07:27:51.519 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:51.515+0000 7fb27b185700 0 -- 172.21.15.5:0/3342963130 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fb2540575c0 msgr2=0x7fb25405dc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:51.748 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:51.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:51.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.72 2024-09-13T07:27:51.914 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:51.911+0000 7f7bcc4fe700 0 -- 172.21.15.5:0/4133592314 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564fb9c3d1c0 msgr2=0x564fb9c50320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:51.938 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:51.943 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:51.943 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.73 2024-09-13T07:27:52.105 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:52.099+0000 7f1f28cff700 0 -- 172.21.15.5:0/3651693506 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1f04008be0 msgr2=0x7f1f0400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:52.132 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:52.137 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:52.137 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.74 2024-09-13T07:27:52.297 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:52.291+0000 7fe13660b700 0 -- 172.21.15.5:0/3453520681 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c383a9d1c0 msgr2=0x55c383ab0390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:52.319 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:52.324 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:52.324 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.75 2024-09-13T07:27:52.492 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:52.487+0000 7f20b74df700 0 -- 172.21.15.5:0/3402685494 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557d01dd47a0 msgr2=0x557d01dd6b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:52.511 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:52.517 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:52.517 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.76 2024-09-13T07:27:52.677 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:52.671+0000 7f0e18792700 0 -- 172.21.15.5:0/1568084593 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e2e206e9d0 msgr2=0x55e2e2084150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:52.703 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:52.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:52.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.77 2024-09-13T07:27:52.869 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:52.863+0000 7fa3053cb700 0 -- 172.21.15.5:0/3317819306 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a6c1d49420 msgr2=0x55a6c1d4b810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:52.896 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:52.891+0000 7fa3053cb700 0 -- 172.21.15.5:0/3317819306 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fa2e4006cb0 msgr2=0x7fa2e803eda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:52.900 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:52.905 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:52.905 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.78 2024-09-13T07:27:53.071 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:53.067+0000 7f093d4f1700 0 -- 172.21.15.5:0/1457985197 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f0928039cc0 msgr2=0x7f092803c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:53.096 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:53.091+0000 7f0946f54700 0 -- 172.21.15.5:0/1457985197 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555b68a1c0d0 msgr2=0x555b68a342c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:53.102 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:53.108 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:53.108 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.79 2024-09-13T07:27:53.269 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:53.263+0000 7fd6b744f700 0 -- 172.21.15.5:0/2711413444 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55923f6ee100 msgr2=0x55923f706310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:27:53.293 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:53.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:53.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.80 2024-09-13T07:27:53.463 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:53.459+0000 7fd9a3bc9700 0 -- 172.21.15.5:0/1326341254 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c6ad1d61c0 msgr2=0x55c6ad1e9320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:53.484 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:53.488 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:53.488 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.81 2024-09-13T07:27:53.647 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:53.643+0000 7f1e476dc700 0 -- 172.21.15.5:0/1457873496 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d07b138c00 msgr2=0x55d07b1f70c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:53.670 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:53.675 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:53.676 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.82 2024-09-13T07:27:53.843 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:53.839+0000 7f2e9470d700 0 -- 172.21.15.5:0/1321738959 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d8f642b9d0 msgr2=0x55d8f6441150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:53.866 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:53.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:53.872 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.83 2024-09-13T07:27:54.033 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.027+0000 7fcc57407700 0 -- 172.21.15.5:0/4204391684 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55676115a100 msgr2=0x556761172350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:54.055 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:54.060 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:54.060 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.84 2024-09-13T07:27:54.229 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.223+0000 7f901bac4700 0 -- 172.21.15.5:0/3127917323 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f900001b3e0 msgr2=0x7f900000b970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:54.251 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:54.257 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:54.257 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.85 2024-09-13T07:27:54.416 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.411+0000 7f38ac47a700 0 -- 172.21.15.5:0/3052458638 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555a3c4340d0 msgr2=0x555a3c44c2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:54.441 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:54.446 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:54.446 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.86 2024-09-13T07:27:54.612 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.607+0000 7f2d85b10700 0 -- 172.21.15.5:0/4060423111 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b263bc49d0 msgr2=0x55b263bdc170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:54.632 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:54.637 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:54.637 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.87 2024-09-13T07:27:54.795 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.791+0000 7fcf42813700 0 -- 172.21.15.5:0/1795248408 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fcf24039cc0 msgr2=0x7fcf2403c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:54.822 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.819+0000 7fcf42813700 0 -- 172.21.15.5:0/1795248408 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d2f197b0d0 msgr2=0x7fcf2403f750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:54.826 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:54.832 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:54.832 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.88 2024-09-13T07:27:54.993 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:54.987+0000 7fb155c12700 0 -- 172.21.15.5:0/3229025584 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fb13c008be0 msgr2=0x7fb13c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.015 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:55.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:55.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.89 2024-09-13T07:27:55.189 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:55.183+0000 7f4bf4c49700 0 -- 172.21.15.5:0/558362984 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f4bd0008be0 msgr2=0x7f4bd000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.213 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:55.218 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:55.218 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.90 2024-09-13T07:27:55.381 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:55.375+0000 7fb5ca9bb700 0 -- 172.21.15.5:0/342148296 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557f22f6f9d0 msgr2=0x557f22f85150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.403 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:55.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:55.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.91 2024-09-13T07:27:55.574 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:55.567+0000 7fa6df4e4700 0 -- 172.21.15.5:0/1496876237 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x556ee30630d0 msgr2=0x556ee307b2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.596 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:55.602 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:55.602 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.92 2024-09-13T07:27:55.761 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:55.755+0000 7fc1381e7700 0 -- 172.21.15.5:0/694917572 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fc11c039d90 msgr2=0x7fc11c03c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.785 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:55.779+0000 7fc136784700 0 -- 172.21.15.5:0/694917572 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fc11400d190 msgr2=0x7fc11400ab10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.793 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:55.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:55.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.93 2024-09-13T07:27:55.961 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:55.955+0000 7fdf03bfc700 0 -- 172.21.15.5:0/2016906233 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55b372b533d0 msgr2=0x55b372b55860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:55.983 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:55.988 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:55.988 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.94 2024-09-13T07:27:56.154 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:56.151+0000 7f144bce6700 0 -- 172.21.15.5:0/32308303 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f254bcc0d0 msgr2=0x55f254be42c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:56.177 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:56.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:56.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.95 2024-09-13T07:27:56.354 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:56.351+0000 7f4527168700 0 -- 172.21.15.5:0/1474289897 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562fc2805100 msgr2=0x562fc281d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:56.381 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:56.386 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:56.386 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.96 2024-09-13T07:27:56.556 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:56.551+0000 7f40330d1700 0 -- 172.21.15.5:0/3460266843 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e9ea70d100 msgr2=0x55e9ea725350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:56.580 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:56.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:56.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.97 2024-09-13T07:27:56.752 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:56.747+0000 7f985a184700 0 -- 172.21.15.5:0/38841641 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562076c931c0 msgr2=0x562076ca6400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:56.776 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:56.781 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:56.782 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.98 2024-09-13T07:27:56.949 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:56.943+0000 7ff4f1bf5700 0 -- 172.21.15.5:0/1662452785 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560a8ba0c240 msgr2=0x560a8ba22440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:56.972 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:56.977 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:56.977 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.99 2024-09-13T07:27:57.132 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:57.127+0000 7ff123acb700 0 -- 172.21.15.5:0/2190734223 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff100008be0 msgr2=0x7ff10000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:57.156 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:27:57.151+0000 7ff1242cc700 0 -- 172.21.15.5:0/2190734223 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55d22fe260e0 msgr2=0x55d22fe28570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:27:57.163 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:27:57.169 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq -w 00 99 2024-09-13T07:27:57.171 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.171 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.00 --image-format 2 -s 1 2024-09-13T07:27:57.256 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.256 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.01 --image-format 2 -s 1 2024-09-13T07:27:57.343 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.343 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.02 --image-format 2 -s 1 2024-09-13T07:27:57.426 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.426 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.03 --image-format 2 -s 1 2024-09-13T07:27:57.513 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.514 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.04 --image-format 2 -s 1 2024-09-13T07:27:57.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.598 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.05 --image-format 2 -s 1 2024-09-13T07:27:57.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.06 --image-format 2 -s 1 2024-09-13T07:27:57.767 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.767 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.07 --image-format 2 -s 1 2024-09-13T07:27:57.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.08 --image-format 2 -s 1 2024-09-13T07:27:57.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:57.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.09 --image-format 2 -s 1 2024-09-13T07:27:58.023 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.023 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.10 --image-format 2 -s 1 2024-09-13T07:27:58.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.11 --image-format 2 -s 1 2024-09-13T07:27:58.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.189 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.12 --image-format 2 -s 1 2024-09-13T07:27:58.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.13 --image-format 2 -s 1 2024-09-13T07:27:58.357 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.357 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.14 --image-format 2 -s 1 2024-09-13T07:27:58.439 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.439 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.15 --image-format 2 -s 1 2024-09-13T07:27:58.527 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.16 --image-format 2 -s 1 2024-09-13T07:27:58.613 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.613 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.17 --image-format 2 -s 1 2024-09-13T07:27:58.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.698 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.18 --image-format 2 -s 1 2024-09-13T07:27:58.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.19 --image-format 2 -s 1 2024-09-13T07:27:58.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.20 --image-format 2 -s 1 2024-09-13T07:27:58.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:58.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.21 --image-format 2 -s 1 2024-09-13T07:27:59.046 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.046 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.22 --image-format 2 -s 1 2024-09-13T07:27:59.134 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.134 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.23 --image-format 2 -s 1 2024-09-13T07:27:59.216 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.217 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.24 --image-format 2 -s 1 2024-09-13T07:27:59.305 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.305 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.25 --image-format 2 -s 1 2024-09-13T07:27:59.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.26 --image-format 2 -s 1 2024-09-13T07:27:59.679 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.679 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.27 --image-format 2 -s 1 2024-09-13T07:27:59.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.28 --image-format 2 -s 1 2024-09-13T07:27:59.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.29 --image-format 2 -s 1 2024-09-13T07:27:59.939 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:27:59.939 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.30 --image-format 2 -s 1 2024-09-13T07:28:00.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.31 --image-format 2 -s 1 2024-09-13T07:28:00.106 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.106 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.32 --image-format 2 -s 1 2024-09-13T07:28:00.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.33 --image-format 2 -s 1 2024-09-13T07:28:00.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.34 --image-format 2 -s 1 2024-09-13T07:28:00.364 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.364 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.35 --image-format 2 -s 1 2024-09-13T07:28:00.450 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.451 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.36 --image-format 2 -s 1 2024-09-13T07:28:00.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.37 --image-format 2 -s 1 2024-09-13T07:28:00.616 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.38 --image-format 2 -s 1 2024-09-13T07:28:00.701 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.701 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.39 --image-format 2 -s 1 2024-09-13T07:28:00.788 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.788 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.40 --image-format 2 -s 1 2024-09-13T07:28:00.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.41 --image-format 2 -s 1 2024-09-13T07:28:00.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:00.959 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.42 --image-format 2 -s 1 2024-09-13T07:28:01.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.044 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.43 --image-format 2 -s 1 2024-09-13T07:28:01.129 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.129 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.44 --image-format 2 -s 1 2024-09-13T07:28:01.217 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.217 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.45 --image-format 2 -s 1 2024-09-13T07:28:01.302 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.302 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.46 --image-format 2 -s 1 2024-09-13T07:28:01.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.47 --image-format 2 -s 1 2024-09-13T07:28:01.474 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.475 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.48 --image-format 2 -s 1 2024-09-13T07:28:01.562 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.562 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.49 --image-format 2 -s 1 2024-09-13T07:28:01.645 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.645 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.50 --image-format 2 -s 1 2024-09-13T07:28:01.735 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.735 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.51 --image-format 2 -s 1 2024-09-13T07:28:01.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.52 --image-format 2 -s 1 2024-09-13T07:28:01.912 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.913 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.53 --image-format 2 -s 1 2024-09-13T07:28:01.998 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:01.998 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.54 --image-format 2 -s 1 2024-09-13T07:28:02.088 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.088 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.55 --image-format 2 -s 1 2024-09-13T07:28:02.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.56 --image-format 2 -s 1 2024-09-13T07:28:02.257 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.257 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.57 --image-format 2 -s 1 2024-09-13T07:28:02.339 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.339 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.58 --image-format 2 -s 1 2024-09-13T07:28:02.423 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.423 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.59 --image-format 2 -s 1 2024-09-13T07:28:02.503 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.503 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.60 --image-format 2 -s 1 2024-09-13T07:28:02.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.590 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.61 --image-format 2 -s 1 2024-09-13T07:28:02.678 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.678 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.62 --image-format 2 -s 1 2024-09-13T07:28:02.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.63 --image-format 2 -s 1 2024-09-13T07:28:02.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.64 --image-format 2 -s 1 2024-09-13T07:28:02.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:02.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.65 --image-format 2 -s 1 2024-09-13T07:28:03.024 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.024 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.66 --image-format 2 -s 1 2024-09-13T07:28:03.107 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.107 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.67 --image-format 2 -s 1 2024-09-13T07:28:03.193 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.193 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.68 --image-format 2 -s 1 2024-09-13T07:28:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.486 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.69 --image-format 2 -s 1 2024-09-13T07:28:03.571 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.571 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.70 --image-format 2 -s 1 2024-09-13T07:28:03.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.71 --image-format 2 -s 1 2024-09-13T07:28:03.736 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.736 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.72 --image-format 2 -s 1 2024-09-13T07:28:03.817 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.817 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.73 --image-format 2 -s 1 2024-09-13T07:28:03.901 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.902 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.74 --image-format 2 -s 1 2024-09-13T07:28:03.983 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:03.983 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.75 --image-format 2 -s 1 2024-09-13T07:28:04.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.76 --image-format 2 -s 1 2024-09-13T07:28:04.154 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.154 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.77 --image-format 2 -s 1 2024-09-13T07:28:04.237 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.237 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.78 --image-format 2 -s 1 2024-09-13T07:28:04.321 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.321 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.79 --image-format 2 -s 1 2024-09-13T07:28:04.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.80 --image-format 2 -s 1 2024-09-13T07:28:04.490 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.490 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.81 --image-format 2 -s 1 2024-09-13T07:28:04.575 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.575 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.82 --image-format 2 -s 1 2024-09-13T07:28:04.656 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.657 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.83 --image-format 2 -s 1 2024-09-13T07:28:04.744 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.745 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.84 --image-format 2 -s 1 2024-09-13T07:28:04.830 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.830 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.85 --image-format 2 -s 1 2024-09-13T07:28:04.918 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:04.918 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.86 --image-format 2 -s 1 2024-09-13T07:28:05.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.87 --image-format 2 -s 1 2024-09-13T07:28:05.085 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.086 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.88 --image-format 2 -s 1 2024-09-13T07:28:05.168 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.168 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.89 --image-format 2 -s 1 2024-09-13T07:28:05.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.90 --image-format 2 -s 1 2024-09-13T07:28:05.339 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.340 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.91 --image-format 2 -s 1 2024-09-13T07:28:05.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.92 --image-format 2 -s 1 2024-09-13T07:28:05.512 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.512 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.93 --image-format 2 -s 1 2024-09-13T07:28:05.593 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.594 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.94 --image-format 2 -s 1 2024-09-13T07:28:05.682 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.682 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.95 --image-format 2 -s 1 2024-09-13T07:28:05.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.96 --image-format 2 -s 1 2024-09-13T07:28:05.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:05.854 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.97 --image-format 2 -s 1 2024-09-13T07:28:06.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:06.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.98 --image-format 2 -s 1 2024-09-13T07:28:06.232 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:06.232 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create image.99 --image-format 2 -s 1 2024-09-13T07:28:06.312 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:06.313 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:06.313 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 100 2024-09-13T07:28:06.365 INFO:tasks.workunit.client.0.smithi005.stdout:100 2024-09-13T07:28:06.366 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:28:06.366 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep image 2024-09-13T07:28:06.366 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:06.367 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 100 2024-09-13T07:28:06.439 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:06.435+0000 7f4b44f6a700 0 -- 172.21.15.5:0/3056677572 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557bb5978070 msgr2=0x557bb598e0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:06.444 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:06.439+0000 7f4b43d08700 0 -- 172.21.15.5:0/3056677572 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f4b2001d010 msgr2=0x7f4b2001f4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:06.718 INFO:tasks.workunit.client.0.smithi005.stdout:100 2024-09-13T07:28:06.719 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq -w 00 99 2024-09-13T07:28:06.720 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:06.720 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.00 2024-09-13T07:28:06.875 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:06.871+0000 7f95eb342700 0 -- 172.21.15.5:0/1313615254 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f95cc039c70 msgr2=0x7f95cc03c080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:06.899 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:06.904 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:06.904 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.01 2024-09-13T07:28:07.065 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.059+0000 7f3e9755d700 0 -- 172.21.15.5:0/708092752 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f3e78039c70 msgr2=0x7f3e7803c080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:07.088 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.083+0000 7f3e9755d700 0 -- 172.21.15.5:0/708092752 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5569cd32b100 msgr2=0x7f3e7803ef70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:07.096 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:07.102 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:07.102 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.02 2024-09-13T07:28:07.273 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.267+0000 7f0b21806700 0 -- 172.21.15.5:0/2564975141 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f0b08008be0 msgr2=0x7f0b0800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:07.294 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:07.299 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:07.299 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.03 2024-09-13T07:28:07.466 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.459+0000 7f880dcda700 0 -- 172.21.15.5:0/331473946 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f87e8005660 msgr2=0x7f87e8008450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:07.491 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:07.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:07.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.04 2024-09-13T07:28:07.661 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.655+0000 7f1e86137700 0 -- 172.21.15.5:0/867267363 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f1e64039d80 msgr2=0x7f1e6403c190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:07.687 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.683+0000 7f1e86137700 0 -- 172.21.15.5:0/867267363 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5629bf8fd0d0 msgr2=0x7f1e6403cbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:07.689 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:07.694 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:07.694 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.05 2024-09-13T07:28:07.860 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:07.855+0000 7f1550b33700 0 -- 172.21.15.5:0/1373205855 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f80e0f72b0 msgr2=0x55f80e0f96a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:07.879 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:07.884 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:07.884 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.06 2024-09-13T07:28:08.041 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:08.035+0000 7f283393e700 0 -- 172.21.15.5:0/278090434 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f281c039cb0 msgr2=0x7f281c03c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:08.066 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:08.063+0000 7f283d3a1700 0 -- 172.21.15.5:0/278090434 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56328edda240 msgr2=0x56328edef430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:08.073 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:08.078 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:08.078 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.07 2024-09-13T07:28:08.459 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:08.455+0000 7efc340fb700 0 -- 172.21.15.5:0/558307600 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55774af2b100 msgr2=0x55774af43350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:08.482 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:08.479+0000 7efc340fb700 0 -- 172.21.15.5:0/558307600 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7efc18039cc0 msgr2=0x7efc1803c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:08.482 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:08.487 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:08.487 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.08 2024-09-13T07:28:08.658 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:08.655+0000 7f2cc6ee9700 0 -- 172.21.15.5:0/2167161786 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55ffb15bd700 msgr2=0x55ffb15bfb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:08.680 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:08.686 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:08.686 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.09 2024-09-13T07:28:08.855 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:08.851+0000 7f260ce1b700 0 -- 172.21.15.5:0/3668370883 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e2dd5f51c0 msgr2=0x55e2dd608400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:08.876 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:08.881 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:08.881 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.10 2024-09-13T07:28:09.055 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:09.051+0000 7fce0934e700 0 -- 172.21.15.5:0/3099550536 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5563bda557a0 msgr2=0x5563bda57b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:09.075 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:09.080 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:09.080 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.11 2024-09-13T07:28:09.245 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:09.239+0000 7fb678846700 0 -- 172.21.15.5:0/2251526388 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fb6540601d0 msgr2=0x7fb65400fbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:09.475 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:09.471+0000 7fb67a2a9700 0 -- 172.21.15.5:0/2251526388 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fb65c039cc0 msgr2=0x7fb65c03c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:09.680 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:09.687 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:09.687 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.12 2024-09-13T07:28:09.854 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:09.847+0000 7efd94723700 0 -- 172.21.15.5:0/1962392643 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c75c6f5430 msgr2=0x55c75c6f7820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:09.878 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:09.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:09.884 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.13 2024-09-13T07:28:10.049 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:10.043+0000 7f7342c38700 0 -- 172.21.15.5:0/1552653144 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x556dadcca0d0 msgr2=0x556dadce22c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:10.071 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:10.077 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:10.077 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.14 2024-09-13T07:28:10.242 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:10.235+0000 7fcfbce41700 0 -- 172.21.15.5:0/1379802364 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555652c6b100 msgr2=0x555652c83350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:10.267 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:10.263+0000 7fcfbce41700 0 -- 172.21.15.5:0/1379802364 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555652c6b100 msgr2=0x7fcfa003f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:10.268 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:10.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:10.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.15 2024-09-13T07:28:10.444 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:10.439+0000 7ff0d1457700 0 -- 172.21.15.5:0/1079625939 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5564ea8b9c00 msgr2=0x5564ea978290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:10.468 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:10.473 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:10.473 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.16 2024-09-13T07:28:10.649 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:10.643+0000 7f8225594700 0 -- 172.21.15.5:0/298278746 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x558fa58e4340 msgr2=0x558fa58e6730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:10.667 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:10.672 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:10.672 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.17 2024-09-13T07:28:10.844 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:10.839+0000 7fa2fc0d5700 0 -- 172.21.15.5:0/1970757631 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56200dafdf80 msgr2=0x56200db18170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:10.866 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:10.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:10.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.18 2024-09-13T07:28:11.040 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:11.035+0000 7f8849010700 0 -- 172.21.15.5:0/898829545 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56253df66f80 msgr2=0x56253df81170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:11.061 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:11.066 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:11.066 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.19 2024-09-13T07:28:11.233 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:11.227+0000 7fbcef618700 0 -- 172.21.15.5:0/2187856035 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55de18b10100 msgr2=0x55de18b28350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:11.461 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:11.455+0000 7fbcef618700 0 -- 172.21.15.5:0/2187856035 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fbcd0039cc0 msgr2=0x7fbcd003c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:11.462 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:11.469 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:11.469 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.20 2024-09-13T07:28:11.636 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:11.631+0000 7f1ed3fce700 0 -- 172.21.15.5:0/2101774977 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f1eb4039cc0 msgr2=0x7f1eb403c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:11.662 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:11.659+0000 7f1ed3fce700 0 -- 172.21.15.5:0/2101774977 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56163f5b49d0 msgr2=0x7f1eb403ee10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:11.663 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:11.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:11.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.21 2024-09-13T07:28:11.840 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:11.835+0000 7f11f99d5700 0 -- 172.21.15.5:0/3413519127 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5593cb285c00 msgr2=0x5593cb344290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:11.866 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:11.872 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:11.872 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.22 2024-09-13T07:28:12.041 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:12.035+0000 7f175bc51700 0 -- 172.21.15.5:0/3395172874 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1734055900 msgr2=0x7f1734067740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:12.065 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:12.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:12.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.23 2024-09-13T07:28:12.242 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:12.235+0000 7f5768a6c700 0 -- 172.21.15.5:0/4169005615 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b42020e0d0 msgr2=0x55b4202262c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:12.264 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:12.268 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:12.268 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.24 2024-09-13T07:28:12.430 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:12.427+0000 7f6e4fcc7700 0 -- 172.21.15.5:0/322471639 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5653389051f0 msgr2=0x565338907680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:12.453 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:12.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:12.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.25 2024-09-13T07:28:12.621 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:12.615+0000 7fbee2c7b700 0 -- 172.21.15.5:0/1253754429 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5605187cf3f0 msgr2=0x5605187d17e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:12.646 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:12.651 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:12.651 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.26 2024-09-13T07:28:12.823 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:12.819+0000 7f9378827700 0 -- 172.21.15.5:0/936115916 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563eabac7240 msgr2=0x563eabadf670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:12.847 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:12.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:12.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.27 2024-09-13T07:28:13.021 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.015+0000 7f2b0e6d0700 0 -- 172.21.15.5:0/3119486587 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d2674ab420 msgr2=0x55d2674ad810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:13.041 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:13.046 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:13.046 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.28 2024-09-13T07:28:13.214 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.211+0000 7f58c367e700 0 -- 172.21.15.5:0/1604669333 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557a0e6b7240 msgr2=0x557a0e6cc430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:13.240 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:13.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:13.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.29 2024-09-13T07:28:13.415 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.411+0000 7f83ce6b2700 0 -- 172.21.15.5:0/4119916733 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55cfdd5055a0 msgr2=0x55cfdd507a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:13.435 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:13.441 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:13.441 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.30 2024-09-13T07:28:13.603 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.599+0000 7f4c2289d700 0 -- 172.21.15.5:0/3666668199 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f4c04039cc0 msgr2=0x7f4c0403c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:13.631 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.627+0000 7f4c24300700 0 -- 172.21.15.5:0/3666668199 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55edb7555100 msgr2=0x55edb756d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:13.637 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:13.644 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:13.644 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.31 2024-09-13T07:28:13.813 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.807+0000 7f0217673700 0 -- 172.21.15.5:0/1869600413 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x560d7c047f80 msgr2=0x560d7c04a410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:13.833 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:13.838 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:13.838 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.32 2024-09-13T07:28:14.003 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:13.999+0000 7fa1ecccc700 0 -- 172.21.15.5:0/813362321 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d940627630 msgr2=0x55d940629a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:14.028 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:14.032 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:14.033 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.33 2024-09-13T07:28:14.201 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:14.195+0000 7f3b82352700 0 -- 172.21.15.5:0/372239282 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f3b60008be0 msgr2=0x7f3b6000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:14.226 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:14.231 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:14.231 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.34 2024-09-13T07:28:14.400 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:14.395+0000 7faaabe55700 0 -- 172.21.15.5:0/1070469366 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7faa8806aea0 msgr2=0x7faa88013fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:14.631 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:14.635 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:14.635 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.35 2024-09-13T07:28:14.806 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:14.799+0000 7f9f74db4700 0 -- 172.21.15.5:0/2378350911 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5587ca7be270 msgr2=0x5587ca7c0700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:14.830 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:14.836 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:14.836 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.36 2024-09-13T07:28:15.004 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:14.999+0000 7f722bd2b700 0 -- 172.21.15.5:0/4056539433 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f720c059100 msgr2=0x7f720c0579c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:15.028 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:15.034 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:15.034 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.37 2024-09-13T07:28:15.191 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:15.187+0000 7f9cccb80700 0 -- 172.21.15.5:0/134886703 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f9cb0039c60 msgr2=0x7f9cb003c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:15.218 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:15.215+0000 7f9cce5e3700 0 -- 172.21.15.5:0/134886703 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x556c800c91c0 msgr2=0x556c800dc320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:15.225 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:15.229 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:15.229 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.38 2024-09-13T07:28:15.398 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:15.395+0000 7f31dab22700 0 -- 172.21.15.5:0/2173923603 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f05f0400d0 msgr2=0x55f05f0587b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:15.419 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:15.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:15.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.39 2024-09-13T07:28:15.602 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:15.599+0000 7f4ae9f3e700 0 -- 172.21.15.5:0/1293335699 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x558a9781c430 msgr2=0x558a97831530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:15.624 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:15.629 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:15.629 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.40 2024-09-13T07:28:15.793 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:15.787+0000 7fbd7b939700 0 -- 172.21.15.5:0/117784457 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x561363cf0240 msgr2=0x561363d05430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:15.817 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:15.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:15.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.41 2024-09-13T07:28:15.999 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:15.995+0000 7f497201a700 0 -- 172.21.15.5:0/958382997 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ebd1cfac00 msgr2=0x55ebd1db92e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:16.223 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:16.229 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:16.229 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.42 2024-09-13T07:28:16.381 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:16.375+0000 7f00f2e84700 0 -- 172.21.15.5:0/3260084828 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5644846d31c0 msgr2=0x7f00d403ef10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:16.407 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:16.403+0000 7f00f1c22700 0 -- 172.21.15.5:0/3260084828 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x564484734260 msgr2=0x5644847366f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:16.420 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:16.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:16.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.43 2024-09-13T07:28:16.596 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:16.591+0000 7f82d00b3700 0 -- 172.21.15.5:0/3357667988 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c3935b1100 msgr2=0x55c3935c9350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:16.617 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:16.622 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:16.622 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.44 2024-09-13T07:28:16.786 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:16.783+0000 7f54c7c37700 0 -- 172.21.15.5:0/2124582548 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5644835ddc00 msgr2=0x56448369c2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:16.811 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:16.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:16.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.45 2024-09-13T07:28:16.984 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:16.979+0000 7f236e606700 0 -- 172.21.15.5:0/3654405638 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cde0948240 msgr2=0x55cde095cf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:17.007 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:17.013 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:17.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.46 2024-09-13T07:28:17.182 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:17.179+0000 7f138d86a700 0 -- 172.21.15.5:0/2088205115 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f136c039cc0 msgr2=0x7f136c03c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:17.206 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:17.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:17.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.47 2024-09-13T07:28:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:17.363+0000 7fa46652c700 0 -- 172.21.15.5:0/4088805910 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x555ce5f6f410 msgr2=0x555ce5f718a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:17.395 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:17.401 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:17.401 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.48 2024-09-13T07:28:17.576 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:17.571+0000 7f1275e85700 0 -- 172.21.15.5:0/4141297974 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555ac1ce83a0 msgr2=0x555ac1d00350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:17.598 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:17.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:17.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.49 2024-09-13T07:28:17.780 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:17.771+0000 7f6b40d60700 0 -- 172.21.15.5:0/1460839646 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55eed829af80 msgr2=0x55eed829d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:17.798 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:17.803 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:17.803 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.50 2024-09-13T07:28:17.972 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:17.967+0000 7ffb4b8f7700 0 -- 172.21.15.5:0/2385491590 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x559bdb9f4100 msgr2=0x559bdba0c310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:17.995 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:18.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:18.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.51 2024-09-13T07:28:18.176 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:18.171+0000 7f08399c5700 0 -- 172.21.15.5:0/951704447 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f0820008be0 msgr2=0x7f082000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:18.201 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:18.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:18.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.52 2024-09-13T07:28:18.372 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:18.367+0000 7fc34bed4700 0 -- 172.21.15.5:0/363763608 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5643cdd730d0 msgr2=0x5643cdd8b2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:18.397 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:18.402 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:18.402 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.53 2024-09-13T07:28:18.576 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:18.571+0000 7fb9fb39e700 0 -- 172.21.15.5:0/936072952 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fb9d4005e00 msgr2=0x7fb9d4002bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:18.601 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:18.605 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:18.606 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.54 2024-09-13T07:28:18.775 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:18.771+0000 7f14565f7700 0 -- 172.21.15.5:0/2923212305 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f143405d6f0 msgr2=0x7f1434015a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:18.799 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:18.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:18.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.55 2024-09-13T07:28:18.978 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:18.971+0000 7fb5d3c2d700 0 -- 172.21.15.5:0/2489851796 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5574fdc25140 msgr2=0x5574fdc27530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:18.998 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:19.003 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:19.003 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.56 2024-09-13T07:28:19.176 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:19.171+0000 7f85dd0a6700 0 -- 172.21.15.5:0/1083715696 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ca92d240d0 msgr2=0x55ca92d3c2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:19.198 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:19.204 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:19.204 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.57 2024-09-13T07:28:19.380 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:19.375+0000 7f1781232700 0 -- 172.21.15.5:0/1494907477 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f1760039c70 msgr2=0x7f176003c080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:19.408 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:19.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:19.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.58 2024-09-13T07:28:19.585 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:19.579+0000 7fa2efddc700 0 -- 172.21.15.5:0/2415621777 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x556600e8e460 msgr2=0x556600e908f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:19.606 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:19.612 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:19.612 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.59 2024-09-13T07:28:19.786 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:19.783+0000 7f70fef6d700 0 -- 172.21.15.5:0/3969648294 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d68fa2a1c0 msgr2=0x55d68fa3d3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:19.809 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:19.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:19.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.60 2024-09-13T07:28:19.983 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:19.979+0000 7f2473f05700 0 -- 172.21.15.5:0/1374407548 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f2454061e90 msgr2=0x7f2454017bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:20.011 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:20.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:20.018 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.61 2024-09-13T07:28:20.177 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:20.171+0000 7f424261a700 0 -- 172.21.15.5:0/2079626582 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x558de501f100 msgr2=0x558de5037800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:20.203 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:20.199+0000 7f42413b8700 0 -- 172.21.15.5:0/2079626582 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f4224039cc0 msgr2=0x7f422403c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:20.209 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:20.214 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:20.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.62 2024-09-13T07:28:20.383 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:20.379+0000 7f630a1ab700 0 -- 172.21.15.5:0/3250337611 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f62f8008be0 msgr2=0x7f62f800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:20.409 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:20.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:20.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.63 2024-09-13T07:28:20.589 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:20.583+0000 7fdd74ada700 0 -- 172.21.15.5:0/2897594229 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ab5e6e61c0 msgr2=0x55ab5e6f9320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:20.615 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:20.620 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:20.620 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.64 2024-09-13T07:28:20.784 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:20.779+0000 7f8e4e55e700 0 -- 172.21.15.5:0/1431532312 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f8e34008be0 msgr2=0x7f8e3400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:20.809 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:20.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:20.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.65 2024-09-13T07:28:20.988 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:20.983+0000 7fc4f34ab700 0 -- 172.21.15.5:0/1536160374 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fc4d8059e40 msgr2=0x7fc4d805e2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:21.221 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:21.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:21.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.66 2024-09-13T07:28:21.398 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:21.395+0000 7f8b82f49700 0 -- 172.21.15.5:0/756948559 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x559e781450d0 msgr2=0x559e7815d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:21.420 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:21.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:21.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.67 2024-09-13T07:28:21.586 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:21.579+0000 7fe8a5efd700 0 -- 172.21.15.5:0/317522235 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fe88c008be0 msgr2=0x7fe88c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:21.614 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:21.611+0000 7fe8a7960700 0 -- 172.21.15.5:0/317522235 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x556026afac00 msgr2=0x556026bb92e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:21.614 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:21.619 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:21.619 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.68 2024-09-13T07:28:21.776 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:21.771+0000 7f6f8022e700 0 -- 172.21.15.5:0/4145110846 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e6dc957100 msgr2=0x55e6dc96f350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:21.799 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:21.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:21.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.69 2024-09-13T07:28:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:21.975+0000 7fac56f2b700 0 -- 172.21.15.5:0/3189926183 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a6530960d0 msgr2=0x55a6530ae2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:22.008 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:22.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:22.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.70 2024-09-13T07:28:22.180 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:22.175+0000 7fa6f04a3700 0 -- 172.21.15.5:0/1061255295 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5617e02e6100 msgr2=0x5617e02fe310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:22.207 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:22.212 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:22.212 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.71 2024-09-13T07:28:22.382 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:22.379+0000 7f4cd4c71700 0 -- 172.21.15.5:0/1168208208 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f4cb0008be0 msgr2=0x7f4cb000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:22.407 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:22.412 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:22.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.72 2024-09-13T07:28:22.580 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:22.575+0000 7f9799141700 0 -- 172.21.15.5:0/280405175 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x563eea83b340 msgr2=0x563eea83d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:22.604 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:22.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:22.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.73 2024-09-13T07:28:22.773 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:22.767+0000 7f00720b3700 0 -- 172.21.15.5:0/3579350221 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f0058012e40 msgr2=0x7f00580132c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:22.802 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:22.808 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:22.808 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.74 2024-09-13T07:28:22.977 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:22.971+0000 7ff4e8613700 0 -- 172.21.15.5:0/3381179322 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ff4c8039cc0 msgr2=0x7ff4c803c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:23.002 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:22.999+0000 7ff4e8613700 0 -- 172.21.15.5:0/3381179322 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c9bf0b9100 msgr2=0x7ff4c803ed70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:23.006 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:23.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:23.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.75 2024-09-13T07:28:23.178 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:23.171+0000 7fdc7a88d700 0 -- 172.21.15.5:0/3606096940 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x564529aa73e0 msgr2=0x564529aa9870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:23.203 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:23.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:23.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.76 2024-09-13T07:28:23.384 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:23.379+0000 7fac947c6700 0 -- 172.21.15.5:0/760309330 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55db4c462f80 msgr2=0x55db4c47b150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:23.408 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:23.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:23.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.77 2024-09-13T07:28:23.584 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:23.579+0000 7f3c763ad700 0 -- 172.21.15.5:0/105975618 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55fe3bc141c0 msgr2=0x55fe3bc27320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:23.605 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:23.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:23.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.78 2024-09-13T07:28:23.780 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:23.775+0000 7fb0d7a13700 0 -- 172.21.15.5:0/3205833892 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560cf42a27a0 msgr2=0x560cf42a4b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:23.806 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:23.812 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:23.812 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.79 2024-09-13T07:28:23.975 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:23.971+0000 7ff30174a700 0 -- 172.21.15.5:0/672871127 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff2e0008d50 msgr2=0x7ff2e000b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:24.001 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:23.995+0000 7ff3031ad700 0 -- 172.21.15.5:0/672871127 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f97f1ce1c0 msgr2=0x55f97f1e1340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:24.008 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:24.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:24.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.80 2024-09-13T07:28:24.187 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:24.183+0000 7f0bb2396700 0 -- 172.21.15.5:0/1200963789 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f0b9805a310 msgr2=0x7f0b98006390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:24.418 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:24.423 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:24.423 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.81 2024-09-13T07:28:24.588 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:24.583+0000 7f39e1f1a700 0 -- 172.21.15.5:0/216983678 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5637fadbc3a0 msgr2=0x5637fadbe790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:24.614 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:24.620 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:24.620 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.82 2024-09-13T07:28:24.783 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:24.779+0000 7f0ac3642700 0 -- 172.21.15.5:0/392911849 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563b15ff60d0 msgr2=0x563b1600e2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:24.804 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:24.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:24.810 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.83 2024-09-13T07:28:24.980 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:24.975+0000 7fced1443700 0 -- 172.21.15.5:0/202043637 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fceb8008be0 msgr2=0x7fceb800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:25.006 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:25.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:25.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.84 2024-09-13T07:28:25.183 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:25.179+0000 7f80d0fa7700 0 -- 172.21.15.5:0/1420531799 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cf6f80d100 msgr2=0x55cf6f825350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:25.207 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:25.212 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:25.212 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.85 2024-09-13T07:28:25.382 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:25.379+0000 7f0a2dac6700 0 -- 172.21.15.5:0/1589847722 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f0a14008be0 msgr2=0x7f0a1400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:25.410 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:25.415 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:25.416 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.86 2024-09-13T07:28:25.594 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:25.591+0000 7fa9e2bcf700 0 -- 172.21.15.5:0/2126879541 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cb49c0a240 msgr2=0x55cb49c1ef40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:25.616 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:25.621 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:25.621 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.87 2024-09-13T07:28:25.796 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:25.791+0000 7f1d9cd7c700 0 -- 172.21.15.5:0/3052915644 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55efeeb68f80 msgr2=0x55efeeb81150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:25.818 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:25.823 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:25.823 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.88 2024-09-13T07:28:25.988 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:25.983+0000 7f73281e9700 0 -- 172.21.15.5:0/2546568594 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f730c008be0 msgr2=0x7f730c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:26.014 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:26.019 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:26.019 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.89 2024-09-13T07:28:26.186 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:26.183+0000 7f3cfb41b700 0 -- 172.21.15.5:0/1694394966 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a30040bc00 msgr2=0x55a3004ca0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:26.217 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:26.221 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:26.222 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.90 2024-09-13T07:28:26.386 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:26.379+0000 7fd84fa83700 0 -- 172.21.15.5:0/3918857874 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562989807100 msgr2=0x56298981f350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:26.416 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:26.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:26.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.91 2024-09-13T07:28:26.591 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:26.587+0000 7f5b5b75e700 0 -- 172.21.15.5:0/2872852530 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f5b40008be0 msgr2=0x7f5b4000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:26.616 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:26.622 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:26.622 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.92 2024-09-13T07:28:26.997 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:26.990+0000 7f183f384700 0 -- 172.21.15.5:0/4001695496 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x565399f40f80 msgr2=0x565399f5b660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.025 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:27.018+0000 7f183f384700 0 -- 172.21.15.5:0/4001695496 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f1820039d90 msgr2=0x7f182003c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.032 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:27.037 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:27.037 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.93 2024-09-13T07:28:27.199 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:27.194+0000 7faaf7dbe700 0 -- 172.21.15.5:0/3316477371 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a2a5bcef80 msgr2=0x55a2a5be9170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.228 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:27.222+0000 7faaf635b700 0 -- 172.21.15.5:0/3316477371 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7faadc039cc0 msgr2=0x7faadc03c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.234 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:27.239 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.94 2024-09-13T07:28:27.416 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:27.410+0000 7f9b49bb5700 0 -- 172.21.15.5:0/808909637 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f9b24007f70 msgr2=0x7f9b24005bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.441 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:27.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:27.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.95 2024-09-13T07:28:27.622 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:27.614+0000 7fbd6761e700 0 -- 172.21.15.5:0/3606215411 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564641801f80 msgr2=0x56464181c660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.643 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:27.648 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:27.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.96 2024-09-13T07:28:27.819 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:27.814+0000 7f6b2f5aa700 0 -- 172.21.15.5:0/604527244 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b88bf2d0d0 msgr2=0x55b88bf452c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:27.845 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:27.850 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:27.850 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.97 2024-09-13T07:28:28.017 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:28.010+0000 7ff21cb0d700 0 -- 172.21.15.5:0/3143848916 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff1f8017670 msgr2=0x7ff1f800cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:28.245 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:28.238+0000 7ff21e570700 0 -- 172.21.15.5:0/3143848916 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5603221139d0 msgr2=0x560322129150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:28.263 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:28.263 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:28.264 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.98 2024-09-13T07:28:28.416 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:28.410+0000 7f6ea8bfd700 0 -- 172.21.15.5:0/4174109726 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f6e88039cb0 msgr2=0x7f6e8803c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:28.446 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:28.442+0000 7f6ea8bfd700 0 -- 172.21.15.5:0/4174109726 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b71cc7a830 msgr2=0x7f6e8803f010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:28.447 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:28.452 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in $(seq -w 00 99) 2024-09-13T07:28:28.452 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm image.99 2024-09-13T07:28:28.625 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:28.618+0000 7f3ce5f28700 0 -- 172.21.15.5:0/2712567435 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563dc03520d0 msgr2=0x563dc036a2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:28.647 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:28.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_remove 2024-09-13T07:28:28.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing remove...' 2024-09-13T07:28:28.654 INFO:tasks.workunit.client.0.smithi005.stdout:testing remove... 2024-09-13T07:28:28.655 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:28:28.655 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:28.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:28.932 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.075 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.214 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.355 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.500 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.647 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.790 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:29.929 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.071 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.357 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.507 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.650 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.793 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:30.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:31.080 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:31.220 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd remove NOT_EXIST 2024-09-13T07:28:31.298 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 0% complete...failed. 2024-09-13T07:28:31.299 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T07:28:31.304 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:28:31.305 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 1 test1 2024-09-13T07:28:31.342 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:28:31.355 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:31.350+0000 7f0edbdc0780 -1 librbd: Forced V1 image creation. 2024-09-13T07:28:31.367 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:28:31.440 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:31.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:31.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:31.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:31.500 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:31.500 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:28:31.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:28:31.761 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:31.754+0000 7f18ad239700 0 -- 172.21.15.5:0/2983002618 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55bb8feca100 msgr2=0x55bb8fee2310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:31.787 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:31.793 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:31.793 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:31.793 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:31.843 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:31.844 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 1 test1 2024-09-13T07:28:31.878 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:28:31.891 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:31.886+0000 7faa496a0780 -1 librbd: Forced V1 image creation. 2024-09-13T07:28:31.902 INFO:tasks.workunit.client.0.smithi005.stderr:+ rados rm -p rbd test1.rbd 2024-09-13T07:28:31.952 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:28:32.006 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:32.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:32.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:32.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:32.263 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:32.264 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 -eq 0 ']' 2024-09-13T07:28:32.264 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:28:32.350 INFO:tasks.workunit.client.0.smithi005.stderr:++ rados -p rbd ls 2024-09-13T07:28:32.351 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep '^rbd_header' 2024-09-13T07:28:32.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ HEADER=rbd_header.18e944742b06 2024-09-13T07:28:32.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ rados -p rbd rm rbd_header.18e944742b06 2024-09-13T07:28:32.468 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:28:32.523 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:32.518+0000 7f8254bce700 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (2) No such file or directory 2024-09-13T07:28:32.528 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:32.522+0000 7f8247fff700 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (2) No such file or directory 2024-09-13T07:28:32.553 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:32.546+0000 7f8247fff700 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (2) No such file or directory 2024-09-13T07:28:32.588 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:32.594 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:32.594 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:32.595 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:32.646 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:32.647 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:28:32.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ rados -p rbd rm rbd_id.test2 2024-09-13T07:28:32.991 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:28:33.163 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:33.158+0000 7f415599b700 0 -- 172.21.15.5:0/1387934826 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564fbcae70d0 msgr2=0x564fbcaff2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:33.189 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:33.194 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:33.194 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:33.195 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:33.240 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:33.240 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:28:33.329 INFO:tasks.workunit.client.0.smithi005.stderr:++ rados -p rbd ls 2024-09-13T07:28:33.329 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep '^rbd_header' 2024-09-13T07:28:33.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ HEADER=rbd_header.190739db7c17 2024-09-13T07:28:33.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ rados -p rbd rm rbd_header.190739db7c17 2024-09-13T07:28:33.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ rados -p rbd rm rbd_id.test2 2024-09-13T07:28:33.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:28:33.555 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:33.550+0000 7f76977fe700 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (2) No such file or directory 2024-09-13T07:28:33.559 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:33.554+0000 7f7697fff700 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (2) No such file or directory 2024-09-13T07:28:33.582 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:33.578+0000 7f7697fff700 -1 librbd::image::OpenRequest: failed to retrieve initial metadata: (2) No such file or directory 2024-09-13T07:28:33.619 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:33.624 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:33.625 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:33.625 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:33.675 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:33.676 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:28:33.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test2@snap 2024-09-13T07:28:34.434 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:28:34.450 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect test2@snap 2024-09-13T07:28:34.572 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test2@snap clone --rbd-default-clone-format 1 2024-09-13T07:28:34.701 INFO:tasks.workunit.client.0.smithi005.stderr:+ rados -p rbd rm rbd_children 2024-09-13T07:28:34.756 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm clone 2024-09-13T07:28:34.929 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:34.922+0000 7fc116084700 0 -- 172.21.15.5:0/525918923 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fc0fc008be0 msgr2=0x7fc0fc00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:34.956 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:34.962 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:34.963 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep clone 2024-09-13T07:28:34.963 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:34.963 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:35.015 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:35.016 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect test2@snap 2024-09-13T07:28:35.140 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm test2@snap 2024-09-13T07:28:35.429 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:28:35.444 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:28:35.617 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:35.610+0000 7f83cf8fe700 0 -- 172.21.15.5:0/3462362667 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5578a073d100 msgr2=0x5578a0755350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:35.647 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:35.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_migration 2024-09-13T07:28:35.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing migration...' 2024-09-13T07:28:35.654 INFO:tasks.workunit.client.0.smithi005.stdout:testing migration... 2024-09-13T07:28:35.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:28:35.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:35.795 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:35.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.355 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.497 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.639 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.780 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:36.925 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.071 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.359 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.501 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.647 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.790 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:37.928 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:38.074 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:28:38.216 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:28:39.536 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:28:39.545 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:28:41.637 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 1 -s 128M test1 2024-09-13T07:28:41.669 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:28:41.682 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:41.682+0000 7fe52f382780 -1 librbd: Forced V1 image creation. 2024-09-13T07:28:41.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:28:41.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'format: 1' 2024-09-13T07:28:41.757 INFO:tasks.workunit.client.0.smithi005.stdout: format: 1 2024-09-13T07:28:41.757 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test1 --image-format 2 2024-09-13T07:28:41.879 INFO:tasks.workunit.client.0.smithi005.stderr:++ get_migration_state test1 2024-09-13T07:28:41.880 INFO:tasks.workunit.client.0.smithi005.stderr:++ local image=test1 2024-09-13T07:28:41.880 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd --format xml status test1 2024-09-13T07:28:41.880 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:42.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ test prepared = prepared 2024-09-13T07:28:42.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:28:42.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'format: 2' 2024-09-13T07:28:42.100 INFO:tasks.workunit.client.0.smithi005.stdout: format: 2 2024-09-13T07:28:42.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:28:42.165 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:42.162+0000 7fbcc394f780 -1 librbd::image::PreRemoveRequest: 0x55ba8e3368e0 validate_image_removal: image in migration state - not removing 2024-09-13T07:28:42.169 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 0% complete...failed. 2024-09-13T07:28:42.169 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: error: image still has watchers 2024-09-13T07:28:42.169 INFO:tasks.workunit.client.0.smithi005.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-13T07:28:42.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:28:42.175 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test1 2024-09-13T07:28:42.308 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:28:42.306+0000 7f3d643e8700 0 -- 172.21.15.5:0/3928354907 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f3d3c004bb0 msgr2=0x7f3d3c006fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:42.316 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:42.314+0000 7f3d6564a700 0 -- 172.21.15.5:0/3928354907 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560a772a01a0 msgr2=0x560a77307dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:42.323 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:42.328 INFO:tasks.workunit.client.0.smithi005.stderr:++ get_migration_state test1 2024-09-13T07:28:42.328 INFO:tasks.workunit.client.0.smithi005.stderr:++ local image=test1 2024-09-13T07:28:42.328 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd --format xml status test1 2024-09-13T07:28:42.328 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:+ test executed = executed 2024-09-13T07:28:42.435 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 2024-09-13T07:28:42.603 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete...2024-09-13T07:28:42.602+0000 7f695da23700 0 -- 172.21.15.5:0/2060652656 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e65bbc83e0 msgr2=0x55e65bbca7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:42.616 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete...2024-09-13T07:28:42.614+0000 7f695da23700 0 -- 172.21.15.5:0/2060652656 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f694003a3d0 msgr2=0x7f694003c800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:42.631 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2024-09-13T07:28:42.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ get_migration_state test1 2024-09-13T07:28:42.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ local image=test1 2024-09-13T07:28:42.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --format xml status test1 2024-09-13T07:28:42.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:42.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:28:42.698 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:28:42.698 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'features: .*layering' 2024-09-13T07:28:42.760 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:28:42.760 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test1 --image-feature layering,exclusive-lock,object-map,fast-diff,deep-flatten 2024-09-13T07:28:42.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:28:42.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'features: .*layering' 2024-09-13T07:28:43.011 INFO:tasks.workunit.client.0.smithi005.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool, migrating 2024-09-13T07:28:43.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test1 2024-09-13T07:28:43.143 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:28:43.142+0000 7f916c499700 0 -- 172.21.15.5:0/4047946871 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f914c004bb0 msgr2=0x7f914c006fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:43.158 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:43.158+0000 7f916d6fb700 0 -- 172.21.15.5:0/4047946871 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c6732973b0 msgr2=0x55c6732fbf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:43.162 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:43.167 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 2024-09-13T07:28:43.334 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:43.334+0000 7f2574c7f700 0 -- 172.21.15.5:0/1513294561 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557e4b8055a0 msgr2=0x557e4b86d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:43.348 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete...2024-09-13T07:28:43.346+0000 7f2574c7f700 0 -- 172.21.15.5:0/1513294561 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f255403a3e0 msgr2=0x7f255403c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:43.375 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2024-09-13T07:28:43.380 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test1 rbd2/test1 2024-09-13T07:28:43.545 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:43.542+0000 7f46c6af6700 0 -- 172.21.15.5:0/1738846767 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5565fe0481c0 msgr2=0x5565fe0acf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:++ get_migration_state rbd2/test1 2024-09-13T07:28:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:++ local image=rbd2/test1 2024-09-13T07:28:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd --format xml status rbd2/test1 2024-09-13T07:28:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:43.715 INFO:tasks.workunit.client.0.smithi005.stderr:+ test prepared = prepared 2024-09-13T07:28:43.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:28:43.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:28:43.716 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:28:43.767 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:28:43.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p rbd2 ls 2024-09-13T07:28:43.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:28:43.821 INFO:tasks.workunit.client.0.smithi005.stdout:test1 2024-09-13T07:28:43.821 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test1 2024-09-13T07:28:43.950 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:28:43.950+0000 7fd35a33d700 0 -- 172.21.15.5:0/2806788041 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cea94e7ca0 msgr2=0x55cea954ce50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:43.955 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:43.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ get_migration_state rbd2/test1 2024-09-13T07:28:43.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ local image=rbd2/test1 2024-09-13T07:28:43.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd --format xml status rbd2/test1 2024-09-13T07:28:43.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:44.086 INFO:tasks.workunit.client.0.smithi005.stderr:+ test executed = executed 2024-09-13T07:28:44.086 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd2/test1 2024-09-13T07:28:44.158 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:44.158+0000 7f7dcaafb700 -1 librbd::image::PreRemoveRequest: 0x55c7265d32c0 validate_image_removal: image in migration state - not removing 2024-09-13T07:28:44.167 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 0% complete...failed. 2024-09-13T07:28:44.167 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: error: image still has watchers 2024-09-13T07:28:44.167 INFO:tasks.workunit.client.0.smithi005.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-13T07:28:44.172 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:28:44.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 2024-09-13T07:28:44.330 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:44.326+0000 7f6806a9f700 0 -- 172.21.15.5:0/7972627 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f67e803a310 msgr2=0x7f67e803c740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:44.353 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2024-09-13T07:28:44.350+0000 7f6806a9f700 0 -- 172.21.15.5:0/7972627 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b8e31200f0 msgr2=0x7f67e803cd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:44.363 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 100% complete...done. 2024-09-13T07:28:44.369 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd2/ns1 2024-09-13T07:28:44.427 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd2/ns2 2024-09-13T07:28:44.484 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare rbd2/test1 rbd2/ns1/test1 2024-09-13T07:28:44.646 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:44.646+0000 7fe299a8e700 0 -- 172.21.15.5:0/317796545 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe280008d50 msgr2=0x7fe28000b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:44.658 INFO:tasks.workunit.client.0.smithi005.stderr:++ get_migration_state rbd2/ns1/test1 2024-09-13T07:28:44.658 INFO:tasks.workunit.client.0.smithi005.stderr:++ local image=rbd2/ns1/test1 2024-09-13T07:28:44.658 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd --format xml status rbd2/ns1/test1 2024-09-13T07:28:44.658 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:44.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ test prepared = prepared 2024-09-13T07:28:44.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute rbd2/test1 2024-09-13T07:28:44.950 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:28:44.950+0000 7fec02649700 0 -- 172.21.15.5:0/1695349287 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x559cd349c650 msgr2=0x559cd349ea40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:44.955 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:44.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ get_migration_state rbd2/ns1/test1 2024-09-13T07:28:44.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ local image=rbd2/ns1/test1 2024-09-13T07:28:44.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd --format xml status rbd2/ns1/test1 2024-09-13T07:28:44.961 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //status/migration/state 2024-09-13T07:28:45.083 INFO:tasks.workunit.client.0.smithi005.stderr:+ test executed = executed 2024-09-13T07:28:45.084 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit rbd2/test1 2024-09-13T07:28:45.245 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2024-09-13T07:28:45.242+0000 7fbb49c5d700 0 -- 172.21.15.5:0/61989607 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d61903a790 msgr2=0x55d61903cb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:45.261 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 100% complete...done. 2024-09-13T07:28:45.267 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare rbd2/ns1/test1 rbd2/ns2/test1 2024-09-13T07:28:45.430 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:45.430+0000 7efc168ab700 0 -- 172.21.15.5:0/1407015630 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x562e190f4390 msgr2=0x562e19109f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:45.443 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute rbd2/ns2/test1 2024-09-13T07:28:45.582 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:28:45.582+0000 7f390fce2700 0 -- 172.21.15.5:0/1828340809 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x555d57ce6ad0 msgr2=0x555d57ce8ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:45.594 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:45.594+0000 7f390fce2700 0 -- 172.21.15.5:0/1828340809 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f38f403a4b0 msgr2=0x7f38f403c8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:45.596 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:45.601 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit rbd2/ns2/test1 2024-09-13T07:28:45.769 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:45.766+0000 7f556e469700 0 -- 172.21.15.5:0/76875660 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5594da842f30 msgr2=0x5594da845320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:45.785 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:45.782+0000 7f556f6cb700 0 -- 172.21.15.5:0/76875660 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5594da7dc0f0 msgr2=0x5594da7f29f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:45.810 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2024-09-13T07:28:45.817 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M test1 2024-09-13T07:28:45.901 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test1 --data-pool rbd2 2024-09-13T07:28:46.079 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:46.078+0000 7fa1898c1700 0 -- 172.21.15.5:0/4143489783 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e24fc53480 msgr2=0x55e24fcb5210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:46.090 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:46.090+0000 7fa1898c1700 0 -- 172.21.15.5:0/4143489783 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fa16803a3e0 msgr2=0x7fa16803c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:46.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:28:46.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'data_pool: rbd2' 2024-09-13T07:28:46.199 INFO:tasks.workunit.client.0.smithi005.stdout: data_pool: rbd2 2024-09-13T07:28:46.200 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test1 2024-09-13T07:28:46.550 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete...2024-09-13T07:28:46.550+0000 7fe822207700 0 -- 172.21.15.5:0/1636414450 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55be5881cca0 msgr2=0x55be58882490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:46.556 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:28:46.554+0000 7fe820fa5700 0 -- 172.21.15.5:0/1636414450 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fe804008d50 msgr2=0x7fe80400b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:46.576 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:46.581 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 2024-09-13T07:28:46.753 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:46.750+0000 7f08afa53700 0 -- 172.21.15.5:0/4250168282 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f0894008be0 msgr2=0x7f089400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:46.770 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:46.766+0000 7f08b0cb5700 0 -- 172.21.15.5:0/4250168282 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55dc143e4b60 msgr2=0x55dc1444d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:46.818 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2024-09-13T07:28:46.825 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test1 2024-09-13T07:28:46.998 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:46.998+0000 7f87cac34700 0 -- 172.21.15.5:0/1129967348 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557e83bc5d70 msgr2=0x557e83c2abb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:47.013 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.010+0000 7f87c99d2700 0 -- 172.21.15.5:0/1129967348 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f87a8008be0 msgr2=0x7f87a800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:47.046 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash mv test1 2024-09-13T07:28:47.047 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv test1 2024-09-13T07:28:47.122 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.122+0000 7f4667a7b780 -1 librbd::api::Trash: move: cannot move migrating image to trash 2024-09-13T07:28:47.132 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: deferred delete error: (16) Device or resource busy 2024-09-13T07:28:47.138 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:28:47.139 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls -a 2024-09-13T07:28:47.139 INFO:tasks.workunit.client.0.smithi005.stderr:++ cut -d ' ' -f 1 2024-09-13T07:28:47.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=1a137fcf213b 2024-09-13T07:28:47.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash rm 1a137fcf213b 2024-09-13T07:28:47.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm 1a137fcf213b 2024-09-13T07:28:47.250 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.250+0000 7fdae6175700 -1 librbd::image::RefreshRequest: image being migrated 2024-09-13T07:28:47.250 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.250+0000 7fdae5974700 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2024-09-13T07:28:47.250 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.250+0000 7fdae5974700 -1 librbd::ImageState: 0x7fdac0050c40 failed to open image: (30) Read-only file system 2024-09-13T07:28:47.250 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.250+0000 7fdadcff9700 -1 librbd::image::RemoveRequest: 0x7fdac0000b80 handle_open_image: error opening image: (30) Read-only file system 2024-09-13T07:28:47.253 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: remove error: (30) Read-only file system 2024-09-13T07:28:47.253 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 0% complete...failed. 2024-09-13T07:28:47.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:28:47.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash restore 1a137fcf213b 2024-09-13T07:28:47.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash restore 1a137fcf213b 2024-09-13T07:28:47.304 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: restore error: (22) Invalid argument 2024-09-13T07:28:47.305 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.302+0000 7ff772a72780 -1 librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4) 2024-09-13T07:28:47.308 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:28:47.309 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test1 2024-09-13T07:28:47.464 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:28:47.462+0000 7f8f06c0a700 0 -- 172.21.15.5:0/1882124585 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f1eb1c03b0 msgr2=0x55f1eb225eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:47.493 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.490+0000 7f8f059a8700 0 -- 172.21.15.5:0/1882124585 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f8ee0004be0 msgr2=0x7f8ee0006fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:47.510 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:28:47.516 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd remove test1 2024-09-13T07:28:47.692 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:28:47.690+0000 7f3e6a9f8700 0 -- 172.21.15.5:0/703874030 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ed4dd32c00 msgr2=0x55ed4ddf1150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:47.717 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:28:47.723 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/dev/urandom bs=1M count=1 2024-09-13T07:28:47.723 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd --image-format 2 import - test1 2024-09-13T07:28:47.818 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-09-13T07:28:47.818 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-09-13T07:28:47.818 INFO:tasks.workunit.client.0.smithi005.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913622 s, 11.5 MB/s 2024-09-13T07:28:47.871 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.870+0000 7fbcc739c700 0 -- 172.21.15.5:0/2332588171 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5609ed94e710 msgr2=0x5609ed950ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:47.885 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:47.882+0000 7fbcc739c700 0 -- 172.21.15.5:0/2332588171 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fbcac03a4b0 msgr2=0x7fbcac03c8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:47.927 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:28:47.934 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export test1 - 2024-09-13T07:28:47.934 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:28:47.999 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:28:48.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ md5sum='0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:28:48.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@snap1 2024-09-13T07:28:48.662 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:28:48.679 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect test1@snap1 2024-09-13T07:28:48.796 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@snap2 2024-09-13T07:28:49.668 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:28:49.686 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test1@snap1 clone_v1 --rbd_default_clone_format=1 2024-09-13T07:28:49.818 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test1@snap2 clone_v2 --rbd_default_clone_format=2 2024-09-13T07:28:49.948 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v1 2024-09-13T07:28:49.949 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'parent: rbd/test1@snap1' 2024-09-13T07:28:50.026 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd/test1@snap1 2024-09-13T07:28:50.026 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v2 2024-09-13T07:28:50.026 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'parent: rbd/test1@snap2' 2024-09-13T07:28:50.262 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:50.262+0000 7f690889b700 0 --2- 172.21.15.5:0/124160033 >> v2:172.21.15.5:3300/0 conn(0x55e51c380490 0x55e51c36c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:28:50.317 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd/test1@snap2 2024-09-13T07:28:50.317 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v2 2024-09-13T07:28:50.318 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'op_features: clone-child' 2024-09-13T07:28:50.398 INFO:tasks.workunit.client.0.smithi005.stdout: op_features: clone-child 2024-09-13T07:28:50.400 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export clone_v1 - 2024-09-13T07:28:50.400 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:28:50.487 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:28:50.493 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '0b687881799fd0eb20a40d70a2490ae4 -' = '0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:28:50.494 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export clone_v2 - 2024-09-13T07:28:50.494 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:28:50.580 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:28:50.587 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '0b687881799fd0eb20a40d70a2490ae4 -' = '0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:28:50.587 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd children test1@snap1 2024-09-13T07:28:50.672 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2024-09-13T07:28:50.673 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd children test1@snap2 2024-09-13T07:28:50.745 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2024-09-13T07:28:50.745 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test1 rbd2/test2 2024-09-13T07:28:50.909 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:50.906+0000 7f8e01004700 0 -- 172.21.15.5:0/2368065170 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d0f32661c0 msgr2=0x55d0f32caf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:50.924 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:50.922+0000 7f8e01004700 0 -- 172.21.15.5:0/2368065170 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f8de403a360 msgr2=0x7f8de403c790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:28:52.815 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v1 2024-09-13T07:28:52.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'parent: rbd2/test2@snap1' 2024-09-13T07:28:52.929 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd2/test2@snap1 2024-09-13T07:28:52.929 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v2 2024-09-13T07:28:52.929 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'parent: rbd2/test2@snap2' 2024-09-13T07:28:53.046 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd2/test2@snap2 2024-09-13T07:28:53.047 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v2 2024-09-13T07:28:53.047 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'op_features: clone-child' 2024-09-13T07:28:53.162 INFO:tasks.workunit.client.0.smithi005.stdout: op_features: clone-child 2024-09-13T07:28:53.163 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd children rbd2/test2@snap1 2024-09-13T07:28:53.284 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2024-09-13T07:28:53.284 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd children rbd2/test2@snap2 2024-09-13T07:28:53.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2024-09-13T07:28:53.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test1 2024-09-13T07:28:53.549 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:53.546+0000 7f7d9d97e700 0 -- 172.21.15.5:0/1774186002 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f7d78004de0 msgr2=0x7f7d780071d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:53.562 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:53.568 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd migration commit test1 2024-09-13T07:28:53.568 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 2024-09-13T07:28:53.708 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: the image has descendants: 2024-09-13T07:28:53.708 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v1 2024-09-13T07:28:53.708 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v2 2024-09-13T07:28:53.708 INFO:tasks.workunit.client.0.smithi005.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-09-13T07:28:53.708 INFO:tasks.workunit.client.0.smithi005.stderr:Ensure no descendant images are opened read-only and run again with force flag. 2024-09-13T07:28:53.714 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:28:53.714 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 --force 2024-09-13T07:28:53.865 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: the image has descendants: 2024-09-13T07:28:53.865 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v1 2024-09-13T07:28:53.865 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v2 2024-09-13T07:28:53.865 INFO:tasks.workunit.client.0.smithi005.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-09-13T07:28:53.865 INFO:tasks.workunit.client.0.smithi005.stderr:Proceeding anyway due to force flag set. 2024-09-13T07:28:53.884 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:53.882+0000 7fcc89357700 0 -- 172.21.15.5:0/2819208742 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55da245354c0 msgr2=0x55da245378b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:53.910 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:53.910+0000 7fcc880f5700 0 -- 172.21.15.5:0/2819208742 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fcc68004de0 msgr2=0x7fcc680071d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:55.740 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 100% complete...done. 2024-09-13T07:28:55.747 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export clone_v1 - 2024-09-13T07:28:55.747 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:28:55.826 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:28:55.833 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '0b687881799fd0eb20a40d70a2490ae4 -' = '0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:28:55.834 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export clone_v2 - 2024-09-13T07:28:55.834 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:28:55.922 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:28:55.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '0b687881799fd0eb20a40d70a2490ae4 -' = '0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:28:55.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare rbd2/test2 test1 2024-09-13T07:28:56.095 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:56.094+0000 7f0b5825b700 0 -- 172.21.15.5:0/3850298394 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f0b380046c0 msgr2=0x7f0b38004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:56.118 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:56.118+0000 7f0b5825b700 0 -- 172.21.15.5:0/3850298394 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f0b3c03a3d0 msgr2=0x7f0b3c03c800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:57.853 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v1 2024-09-13T07:28:57.853 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'parent: rbd/test1@snap1' 2024-09-13T07:28:57.967 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd/test1@snap1 2024-09-13T07:28:57.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v2 2024-09-13T07:28:57.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'parent: rbd/test1@snap2' 2024-09-13T07:28:58.083 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd/test1@snap2 2024-09-13T07:28:58.083 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info clone_v2 2024-09-13T07:28:58.084 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 'op_features: clone-child' 2024-09-13T07:28:58.201 INFO:tasks.workunit.client.0.smithi005.stdout: op_features: clone-child 2024-09-13T07:28:58.201 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd children test1@snap1 2024-09-13T07:28:58.316 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2024-09-13T07:28:58.317 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd children test1@snap2 2024-09-13T07:28:58.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2024-09-13T07:28:58.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test1 2024-09-13T07:28:58.586 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:58.586+0000 7f558f58a700 0 -- 172.21.15.5:0/2311938450 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f556c008be0 msgr2=0x7f556c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:58.607 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:58.606+0000 7f558f58a700 0 -- 172.21.15.5:0/2311938450 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f557403a360 msgr2=0x7f557403c790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:58.627 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:28:58.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd migration commit test1 2024-09-13T07:28:58.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 2024-09-13T07:28:58.789 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:58.786+0000 7fedd7ee5700 0 -- 172.21.15.5:0/559899703 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56442fea95a0 msgr2=0x56442ff114a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:58.791 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: the image has descendants: 2024-09-13T07:28:58.791 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v1 2024-09-13T07:28:58.791 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v2 2024-09-13T07:28:58.791 INFO:tasks.workunit.client.0.smithi005.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-09-13T07:28:58.791 INFO:tasks.workunit.client.0.smithi005.stderr:Ensure no descendant images are opened read-only and run again with force flag. 2024-09-13T07:28:58.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:28:58.799 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration commit test1 --force 2024-09-13T07:28:58.966 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:58.966+0000 7f475a090700 0 -- 172.21.15.5:0/3413191744 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x565498b18130 msgr2=0x565498b7a390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:28:58.968 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: the image has descendants: 2024-09-13T07:28:58.968 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v1 2024-09-13T07:28:58.968 INFO:tasks.workunit.client.0.smithi005.stderr: rbd/clone_v2 2024-09-13T07:28:58.968 INFO:tasks.workunit.client.0.smithi005.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2024-09-13T07:28:58.968 INFO:tasks.workunit.client.0.smithi005.stderr:Proceeding anyway due to force flag set. 2024-09-13T07:28:58.997 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:28:58.994+0000 7f475a090700 0 -- 172.21.15.5:0/3413191744 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f473803a290 msgr2=0x7f473803c6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:00.757 INFO:tasks.workunit.client.0.smithi005.stderr: Commit image migration: 100% complete...done. 2024-09-13T07:29:00.764 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export clone_v1 - 2024-09-13T07:29:00.764 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:29:00.848 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:29:00.855 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '0b687881799fd0eb20a40d70a2490ae4 -' = '0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:29:00.856 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export clone_v2 - 2024-09-13T07:29:00.856 INFO:tasks.workunit.client.0.smithi005.stderr:++ md5sum 2024-09-13T07:29:00.937 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:29:00.946 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '0b687881799fd0eb20a40d70a2490ae4 -' = '0b687881799fd0eb20a40d70a2490ae4 -' 2024-09-13T07:29:00.947 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd remove clone_v1 2024-09-13T07:29:01.133 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:01.130+0000 7f5ca8ce4700 0 -- 172.21.15.5:0/755564612 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56370ff4c1c0 msgr2=0x56370ff5f320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:01.158 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:01.163 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd remove clone_v2 2024-09-13T07:29:01.329 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:01.326+0000 7f09fe643700 0 -- 172.21.15.5:0/891198688 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f09dc008be0 msgr2=0x7f09dc00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:01.357 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:01.358+0000 7f0a000a6700 0 -- 172.21.15.5:0/891198688 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5642286d0830 msgr2=0x5642286d2c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:01.365 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:01.370 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect test1@snap1 2024-09-13T07:29:01.492 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge test1 2024-09-13T07:29:02.724 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:29:02.741 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:29:02.921 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:02.918+0000 7f309e07b700 0 -- 172.21.15.5:0/3813761743 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f307c008be0 msgr2=0x7f307c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:03.157 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:03.154+0000 7f309fade700 0 -- 172.21.15.5:0/3813761743 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5561e6caf100 msgr2=0x5561e6cc7350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:03.158 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:03.164 INFO:tasks.workunit.client.0.smithi005.stderr:+ for format in 1 2 2024-09-13T07:29:03.164 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 1 test2 2024-09-13T07:29:03.200 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:29:03.213 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:03.210+0000 7fe307351780 -1 librbd: Forced V1 image creation. 2024-09-13T07:29:03.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-09-13T07:29:03.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:03.429 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:03.436 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:03.437 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:03.446 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:03.567 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:03.566+0000 7ff156824700 0 -- 172.21.15.5:0/974545296 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7ff130004be0 msgr2=0x7ff130006fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:03.592 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:03.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:03.655 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:03.659 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:03.659 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 250.005 bytes/sec: 250 KiB/s 2024-09-13T07:29:03.668 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:03.755 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T07:29:03.759 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 1 test2 2024-09-13T07:29:03.790 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:29:03.800 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:03.798+0000 7f06e5212780 -1 librbd: Forced V1 image creation. 2024-09-13T07:29:03.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-09-13T07:29:03.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:04.013 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:04.020 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:04.021 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:04.030 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test2 2024-09-13T07:29:04.161 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... 2024-09-13T07:29:04.158+0000 7f8bb1c65700 0 -- 172.21.15.5:0/3744763511 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560dd3c3bf80 msgr2=0x560dd3c3e370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:04.167 INFO:tasks.workunit.client.0.smithi005.stderr:Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:29:04.166+0000 7f8bb1c65700 0 -- 172.21.15.5:0/3744763511 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560dd3bd1ca0 msgr2=0x7f8b9403d680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:04.176 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:29:04.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:04.296 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:04.294+0000 7f232118d700 0 -- 172.21.15.5:0/407989509 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f230403a3f0 msgr2=0x7f230403c820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:04.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:04.318+0000 7f2322bf0700 0 -- 172.21.15.5:0/407989509 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x5569641127c0 msgr2=0x556964176f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:04.323 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:04.328 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:04.383 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:04.388 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:04.388 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 250.005 bytes/sec: 250 KiB/s 2024-09-13T07:29:04.395 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:04.480 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T07:29:04.484 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 1 test2 2024-09-13T07:29:04.516 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:29:04.529 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:04.526+0000 7fd0f2400780 -1 librbd: Forced V1 image creation. 2024-09-13T07:29:04.540 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 --data-pool INVALID_DATA_POOL 2024-09-13T07:29:04.616 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:04.614+0000 7eff511aa780 -1 librbd::image::CreateRequest: 0x56234e91fc60 validate_data_pool: data pool INVALID_DATA_POOL does not exist 2024-09-13T07:29:04.616 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:04.614+0000 7eff511aa780 -1 librbd::Migration: create_dst_image: header creation failed: (2) No such file or directory 2024-09-13T07:29:04.629 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: preparing migration failed: (2) No such file or directory 2024-09-13T07:29:04.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:29:04.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:04.693 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:04.902 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:04.903 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 4.71708 bytes/sec: 4.7 KiB/s 2024-09-13T07:29:04.913 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:04.998 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T07:29:05.002 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 1 test2 2024-09-13T07:29:05.034 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:29:05.045 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:05.046+0000 7fa07af88780 -1 librbd: Forced V1 image creation. 2024-09-13T07:29:05.059 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 rbd2/test2 2024-09-13T07:29:05.183 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/test2 2024-09-13T07:29:05.257 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:05.262 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:05.263 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:05.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:05.386 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:05.386+0000 7f788c877700 0 -- 172.21.15.5:0/2934670374 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x562b03f753d0 msgr2=0x562b03fdd350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:05.390 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:05.395 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:05.451 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:05.455 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:05.455 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 250.005 bytes/sec: 250 KiB/s 2024-09-13T07:29:05.463 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:05.549 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T07:29:05.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 1 test2 2024-09-13T07:29:05.584 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image format 1 is deprecated 2024-09-13T07:29:05.595 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:05.594+0000 7fd02b8e9780 -1 librbd: Forced V1 image creation. 2024-09-13T07:29:05.608 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 rbd2/test2 2024-09-13T07:29:05.732 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort rbd2/test2 2024-09-13T07:29:05.851 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:05.850+0000 7fc41ade7700 0 -- 172.21.15.5:0/3511987336 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x5572ca63c2e0 msgr2=0x5572ca69f0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:05.869 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:05.873 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:05.931 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:05.934 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:05.934 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-09-13T07:29:05.942 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:06.026 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T07:29:06.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 = 1 2024-09-13T07:29:06.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ continue 2024-09-13T07:29:06.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ for format in 1 2 2024-09-13T07:29:06.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 2 test2 2024-09-13T07:29:06.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-09-13T07:29:06.287 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:06.286+0000 7f2f00a52700 0 -- 172.21.15.5:0/878498900 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f2ee0004bb0 msgr2=0x7f2ee00075d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:06.300 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:06.298+0000 7f2f01253700 0 -- 172.21.15.5:0/878498900 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f2edc008be0 msgr2=0x7f2edc00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:06.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:06.444 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:06.455 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:06.455 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 83.3351 bytes/sec: 83 KiB/s 2024-09-13T07:29:06.477 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:06.628 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:06.626+0000 7fb158175700 0 -- 172.21.15.5:0/3662571099 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fb13c008d50 msgr2=0x7fb13c00b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:06.652 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:06.650+0000 7fb1593d7700 0 -- 172.21.15.5:0/3662571099 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x55b09d6c8ca0 msgr2=0x55b09d72e490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:06.697 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:06.703 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:06.794 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:06.800 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:06.800 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2024-09-13T07:29:06.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:06.999 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:06.998+0000 7fb5556a0700 0 -- 172.21.15.5:0/3282209428 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a02e025f80 msgr2=0x55a02e040660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:07.032 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:07.041 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 2 test2 2024-09-13T07:29:07.129 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 --data-pool rbd2 2024-09-13T07:29:07.309 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:07.306+0000 7fcc2d726700 0 -- 172.21.15.5:0/3272903771 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fcc08008be0 msgr2=0x7fcc0800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:07.320 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:07.318+0000 7fcc2d726700 0 -- 172.21.15.5:0/3272903771 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fcc1003a420 msgr2=0x7fcc1003c830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:07.357 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:07.460 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:07.470 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:07.471 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 83.3352 bytes/sec: 83 KiB/s 2024-09-13T07:29:07.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration execute test2 2024-09-13T07:29:07.653 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 2024-09-13T07:29:07.650+0000 7f792aa28700 0 -- 172.21.15.5:0/3092990500 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f7908008be0 msgr2=0x7f790800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:07.657 INFO:tasks.workunit.client.0.smithi005.stderr:34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2024-09-13T07:29:07.654+0000 7f792a227700 0 -- 172.21.15.5:0/3092990500 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f790c004bb0 msgr2=0x7f790c007580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:07.681 INFO:tasks.workunit.client.0.smithi005.stderr: Image migration: 100% complete...done. 2024-09-13T07:29:07.686 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:07.843 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:07.842+0000 7f2bd5189700 0 -- 172.21.15.5:0/4032168596 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f2bb0008be0 msgr2=0x7f2bb000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:07.847 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:29:07.846+0000 7f3284771700 -1 reset not still connected to 0x5597c458dba0 2024-09-13T07:29:07.847 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:29:07.846+0000 7f3284771700 -1 reset not still connected to 0x5597c458de10 2024-09-13T07:29:07.869 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:07.870+0000 7f2bd63eb700 0 -- 172.21.15.5:0/4032168596 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x55cfb04141a0 msgr2=0x55cfb047bcd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:07.913 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:07.919 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:08.013 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:08.020 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:08.020 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 250.005 bytes/sec: 250 KiB/s 2024-09-13T07:29:08.040 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:08.223 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:08.222+0000 7f0fc0641700 0 -- 172.21.15.5:0/1784581035 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a823c700d0 msgr2=0x55a823c882c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:08.258 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:08.263 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 2 test2 2024-09-13T07:29:08.353 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 --data-pool INVALID_DATA_POOL 2024-09-13T07:29:08.480 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:08.478+0000 7f1c84200780 -1 librbd::image::CreateRequest: 0x55ac10e86c60 validate_data_pool: data pool INVALID_DATA_POOL does not exist 2024-09-13T07:29:08.481 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:08.478+0000 7f1c84200780 -1 librbd::Migration: create_dst_image: header creation failed: (2) No such file or directory 2024-09-13T07:29:08.507 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:08.506+0000 7f1c83f99700 0 -- 172.21.15.5:0/966103535 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x55ac110bf840 msgr2=0x55ac11124900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:08.535 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:08.534+0000 7f1c83f99700 0 -- 172.21.15.5:0/966103535 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f1c68015b40 msgr2=0x7f1c6403f570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:08.541 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: preparing migration failed: (2) No such file or directory 2024-09-13T07:29:08.546 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:29:08.547 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:08.642 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:08.650 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:08.651 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:08.673 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:08.843 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:08.842+0000 7f7bbe220700 0 -- 172.21.15.5:0/3386398210 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b78ce4d100 msgr2=0x55b78ce65350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:08.878 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:08.878+0000 7f7bbe220700 0 -- 172.21.15.5:0/3386398210 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f7ba0008be0 msgr2=0x7f7b9c03d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:08.887 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:08.893 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 2 test2 2024-09-13T07:29:08.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 rbd2/test2 2024-09-13T07:29:09.150 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:09.150+0000 7fe80d20e700 0 -- 172.21.15.5:0/1113174631 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe7e8008be0 msgr2=0x7fe7e800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:09.160 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:09.158+0000 7fe80d20e700 0 -- 172.21.15.5:0/1113174631 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe7f003a350 msgr2=0x7fe7f003c760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:09.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/test2 2024-09-13T07:29:09.293 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:09.302 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:09.302 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 83.3352 bytes/sec: 83 KiB/s 2024-09-13T07:29:09.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:09.469 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:09.466+0000 7ffb7b476700 0 -- 172.21.15.5:0/1127895902 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7ffb60004bf0 msgr2=0x7ffb60008670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:09.498 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:09.498+0000 7ffb7bc77700 0 -- 172.21.15.5:0/1127895902 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ffb5c03a340 msgr2=0x7ffb5c03c750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:09.521 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:09.527 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:09.620 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:09.627 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:09.627 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:09.648 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:09.819 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:09.818+0000 7f530a417700 0 -- 172.21.15.5:0/1254226837 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f52ec03a350 msgr2=0x7f52ec03c760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:09.850 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:09.850+0000 7f53089b4700 0 -- 172.21.15.5:0/1254226837 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f52e800a1d0 msgr2=0x7f52e80094a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:09.861 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:09.867 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 2 test2 2024-09-13T07:29:09.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 rbd2/test2 2024-09-13T07:29:10.126 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:10.126+0000 7fd4bd0f8700 0 -- 172.21.15.5:0/3858479600 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fd494004bb0 msgr2=0x7fd4940075d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:10.136 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:10.134+0000 7fd4bd8f9700 0 -- 172.21.15.5:0/3858479600 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fd49c008be0 msgr2=0x7fd49c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:10.167 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort rbd2/test2 2024-09-13T07:29:10.323 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:10.322+0000 7fe9e724b700 0 -- 172.21.15.5:0/1601076062 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55558e3b11a0 msgr2=0x55558e418b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:10.347 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:10.346+0000 7fe9e5fe9700 0 -- 172.21.15.5:0/1601076062 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe9c4008be0 msgr2=0x7fe9c400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:10.368 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:10.373 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:10.459 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:10.469 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:10.469 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 83.3352 bytes/sec: 83 KiB/s 2024-09-13T07:29:10.491 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:10.667 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:10.666+0000 7f17d10b0700 0 -- 172.21.15.5:0/3256694528 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b1058e8100 msgr2=0x55b105900350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:10.704 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:10.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 2 = 1 2024-09-13T07:29:10.709 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 128M --image-format 2 test2 2024-09-13T07:29:10.793 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration prepare test2 rbd2/ns1/test3 2024-09-13T07:29:10.967 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:10.966+0000 7fe3f7ff7700 0 -- 172.21.15.5:0/2505275163 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x564c9a4ccee0 msgr2=0x564c9a4cf2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:10.977 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:10.978+0000 7fe3f6594700 0 -- 172.21.15.5:0/2505275163 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe3d4008be0 msgr2=0x7fe3d400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:11.010 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/ns1/test3 2024-09-13T07:29:11.110 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:11.119 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:11.119 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:11.137 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd migration abort test2 2024-09-13T07:29:11.293 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2024-09-13T07:29:11.290+0000 7fc3fd24b700 0 -- 172.21.15.5:0/4020052324 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a40916a060 msgr2=0x55a40918a6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:11.322 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:11.322+0000 7fc3fd24b700 0 -- 172.21.15.5:0/4020052324 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fc3e003a420 msgr2=0x7fc3e003c830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:11.339 INFO:tasks.workunit.client.0.smithi005.stderr: Abort image migration: 100% complete...done. 2024-09-13T07:29:11.344 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2024-09-13T07:29:11.433 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-13T07:29:11.439 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:29:11.439 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2024-09-13T07:29:11.462 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:29:11.640 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:11.638+0000 7fd387782700 0 -- 172.21.15.5:0/3886991155 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55feb683f100 msgr2=0x55feb6857310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:11.673 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:11.678 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:11.679 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:11.818 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:11.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:12.111 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:12.302 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:12.648 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:12.797 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:12.945 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.093 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.241 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.390 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.541 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.691 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.836 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:13.986 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:14.130 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:14.281 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:14.428 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:14.578 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:29:15.746 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:29:15.755 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_config 2024-09-13T07:29:15.755 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing config...' 2024-09-13T07:29:15.755 INFO:tasks.workunit.client.0.smithi005.stdout:testing config... 2024-09-13T07:29:15.756 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:15.756 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:15.886 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.026 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.321 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.469 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:16.901 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.048 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.197 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.477 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.620 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.767 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:17.914 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:18.060 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:18.200 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:18.342 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config global set osd rbd_cache true 2024-09-13T07:29:18.342 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set osd rbd_cache true 2024-09-13T07:29:18.379 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: invalid config entity: osd (must be global, client or client.) 2024-09-13T07:29:18.381 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:18.382 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config global set global debug_ms 10 2024-09-13T07:29:18.382 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set global debug_ms 10 2024-09-13T07:29:18.418 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: not rbd option: debug_ms 2024-09-13T07:29:18.420 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:18.420 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config global set global rbd_UNKNOWN false 2024-09-13T07:29:18.420 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set global rbd_UNKNOWN false 2024-09-13T07:29:18.449 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: invalid config key: rbd_UNKNOWN 2024-09-13T07:29:18.451 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:18.451 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config global set global rbd_cache INVALID 2024-09-13T07:29:18.451 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set global rbd_cache INVALID 2024-09-13T07:29:18.496 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: error setting rbd_cache: error parsing value: Expected option value to be integer, got 'INVALID' 2024-09-13T07:29:18.502 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:18.502 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set global rbd_cache false 2024-09-13T07:29:18.554 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set client rbd_cache true 2024-09-13T07:29:18.604 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global set client.123 rbd_cache false 2024-09-13T07:29:18.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global get global rbd_cache 2024-09-13T07:29:18.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^false$' 2024-09-13T07:29:18.705 INFO:tasks.workunit.client.0.smithi005.stdout:false 2024-09-13T07:29:18.706 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global get client rbd_cache 2024-09-13T07:29:18.706 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^true$' 2024-09-13T07:29:18.752 INFO:tasks.workunit.client.0.smithi005.stdout:true 2024-09-13T07:29:18.752 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global get client.123 rbd_cache 2024-09-13T07:29:18.752 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^false$' 2024-09-13T07:29:18.803 INFO:tasks.workunit.client.0.smithi005.stdout:false 2024-09-13T07:29:18.803 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config global get client.UNKNOWN rbd_cache 2024-09-13T07:29:18.803 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global get client.UNKNOWN rbd_cache 2024-09-13T07:29:18.846 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd_cache is not set 2024-09-13T07:29:18.851 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:18.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global list global 2024-09-13T07:29:18.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * false * global *$' 2024-09-13T07:29:18.898 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache false global 2024-09-13T07:29:18.898 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global list client 2024-09-13T07:29:18.899 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * true * client *$' 2024-09-13T07:29:18.948 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache true client 2024-09-13T07:29:18.949 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global list client.123 2024-09-13T07:29:18.949 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * false * client.123 *$' 2024-09-13T07:29:19.000 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache false client.123 2024-09-13T07:29:19.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global list client.UNKNOWN 2024-09-13T07:29:19.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * true * client *$' 2024-09-13T07:29:19.049 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache true client 2024-09-13T07:29:19.049 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global rm client rbd_cache 2024-09-13T07:29:19.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config global get client rbd_cache 2024-09-13T07:29:19.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global get client rbd_cache 2024-09-13T07:29:19.145 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd_cache is not set 2024-09-13T07:29:19.150 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:19.151 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global list client 2024-09-13T07:29:19.151 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * false * global *$' 2024-09-13T07:29:19.199 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache false global 2024-09-13T07:29:19.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global rm client.123 rbd_cache 2024-09-13T07:29:19.250 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config global rm global rbd_cache 2024-09-13T07:29:19.508 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config pool set rbd rbd_cache true 2024-09-13T07:29:19.579 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config pool list rbd 2024-09-13T07:29:19.579 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * true * pool *$' 2024-09-13T07:29:19.629 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache true pool 2024-09-13T07:29:19.630 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^true$' 2024-09-13T07:29:19.630 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config pool get rbd rbd_cache 2024-09-13T07:29:19.684 INFO:tasks.workunit.client.0.smithi005.stdout:true 2024-09-13T07:29:19.685 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 1 test1 2024-09-13T07:29:19.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image list rbd/test1 2024-09-13T07:29:19.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * true * pool *$' 2024-09-13T07:29:19.845 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache true pool 2024-09-13T07:29:19.845 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image set rbd/test1 rbd_cache false 2024-09-13T07:29:19.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image list rbd/test1 2024-09-13T07:29:19.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * false * image *$' 2024-09-13T07:29:20.045 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache false image 2024-09-13T07:29:20.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image get rbd/test1 rbd_cache 2024-09-13T07:29:20.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^false$' 2024-09-13T07:29:20.117 INFO:tasks.workunit.client.0.smithi005.stdout:false 2024-09-13T07:29:20.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image remove rbd/test1 rbd_cache 2024-09-13T07:29:20.246 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config image get rbd/test1 rbd_cache 2024-09-13T07:29:20.246 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image get rbd/test1 rbd_cache 2024-09-13T07:29:20.305 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd_cache is not set 2024-09-13T07:29:20.314 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:20.314 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config image list rbd/test1 2024-09-13T07:29:20.314 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * true * pool *$' 2024-09-13T07:29:20.390 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache true pool 2024-09-13T07:29:20.390 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config pool remove rbd rbd_cache 2024-09-13T07:29:20.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd config pool get rbd rbd_cache 2024-09-13T07:29:20.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config pool get rbd rbd_cache 2024-09-13T07:29:20.498 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd_cache is not set 2024-09-13T07:29:20.504 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:29:20.505 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd config pool list rbd 2024-09-13T07:29:20.505 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^rbd_cache * true * config *$' 2024-09-13T07:29:20.557 INFO:tasks.workunit.client.0.smithi005.stdout:rbd_cache true config 2024-09-13T07:29:20.557 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:29:20.734 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:20.733+0000 7fb1c79b0700 0 -- 172.21.15.5:0/2900488026 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fb1a0005150 msgr2=0x7fb1a000d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:20.968 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:20.973 INFO:tasks.workunit.client.0.smithi005.stderr:+ RBD_CREATE_ARGS= 2024-09-13T07:29:20.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_others 2024-09-13T07:29:20.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing import, export, resize, and snapshots...' 2024-09-13T07:29:20.974 INFO:tasks.workunit.client.0.smithi005.stdout:testing import, export, resize, and snapshots... 2024-09-13T07:29:20.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ TMP_FILES='/tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1' 2024-09-13T07:29:20.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:20.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.115 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.261 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.701 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.841 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:21.983 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:22.122 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:22.272 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:22.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:22.607 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:22.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:22.909 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:23.054 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:23.201 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:23.352 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:23.504 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:23.647 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-09-13T07:29:23.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/sh of=/tmp/img1 bs=1k count=1 seek=10 2024-09-13T07:29:23.650 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-09-13T07:29:23.650 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-09-13T07:29:23.650 INFO:tasks.workunit.client.0.smithi005.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000169716 s, 6.0 MB/s 2024-09-13T07:29:23.651 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/dd of=/tmp/img1 bs=1k count=10 seek=100 2024-09-13T07:29:23.651 INFO:tasks.workunit.client.0.smithi005.stderr:10+0 records in 2024-09-13T07:29:23.651 INFO:tasks.workunit.client.0.smithi005.stderr:10+0 records out 2024-09-13T07:29:23.651 INFO:tasks.workunit.client.0.smithi005.stderr:10240 bytes (10 kB, 10 KiB) copied, 0.000104672 s, 97.8 MB/s 2024-09-13T07:29:23.652 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/rm of=/tmp/img1 bs=1k count=100 seek=1000 2024-09-13T07:29:23.653 INFO:tasks.workunit.client.0.smithi005.stderr:70+1 records in 2024-09-13T07:29:23.653 INFO:tasks.workunit.client.0.smithi005.stderr:70+1 records out 2024-09-13T07:29:23.653 INFO:tasks.workunit.client.0.smithi005.stderr:72056 bytes (72 kB, 70 KiB) copied, 0.000268794 s, 268 MB/s 2024-09-13T07:29:23.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/ls of=/tmp/img1 bs=1k seek=10000 2024-09-13T07:29:23.656 INFO:tasks.workunit.client.0.smithi005.stderr:138+1 records in 2024-09-13T07:29:23.656 INFO:tasks.workunit.client.0.smithi005.stderr:138+1 records out 2024-09-13T07:29:23.656 INFO:tasks.workunit.client.0.smithi005.stderr:142144 bytes (142 kB, 139 KiB) copied, 0.00104774 s, 136 MB/s 2024-09-13T07:29:23.656 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/ln of=/tmp/img1 bs=1k seek=100000 2024-09-13T07:29:23.657 INFO:tasks.workunit.client.0.smithi005.stderr:74+1 records in 2024-09-13T07:29:23.657 INFO:tasks.workunit.client.0.smithi005.stderr:74+1 records out 2024-09-13T07:29:23.657 INFO:tasks.workunit.client.0.smithi005.stderr:76160 bytes (76 kB, 74 KiB) copied, 0.000270611 s, 281 MB/s 2024-09-13T07:29:23.657 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import /tmp/img1 testimg1 2024-09-13T07:29:23.855 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 4% complete... Importing image: 8% complete... Importing image: 12% complete... Importing image: 16% complete... Importing image: 20% complete... Importing image: 24% complete... Importing image: 28% complete... Importing image: 32% complete... Importing image: 36% complete... Importing image: 40% complete... Importing image: 45% complete... Importing image: 49% complete... Importing image: 53% complete... Importing image: 57% complete... Importing image: 61% complete... Importing image: 65% complete... Importing image: 69% complete... Importing image: 73% complete... Importing image: 77% complete... Importing image: 81% complete... Importing image: 85% complete... Importing image: 90% complete... Importing image: 94% complete... Importing image: 98% complete...2024-09-13T07:29:23.853+0000 7fca31f21700 0 -- 172.21.15.5:0/2464067771 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fca10039cf0 msgr2=0x7fca1003c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:23.858 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:29:23.864 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd resize testimg1 --size=256 --allow-shrink 2024-09-13T07:29:24.176 INFO:tasks.workunit.client.0.smithi005.stderr: Resizing image: 100% complete...done. 2024-09-13T07:29:24.193 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img2 2024-09-13T07:29:24.315 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:24.319 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1 --snap=snap1 2024-09-13T07:29:24.606 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:29:24.623 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd resize testimg1 --size=128 2024-09-13T07:29:24.682 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: shrinking an image is only allowed with the --allow-shrink flag 2024-09-13T07:29:24.690 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:29:24.691 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd resize testimg1 --size=128 --allow-shrink 2024-09-13T07:29:24.799 INFO:tasks.workunit.client.0.smithi005.stderr: Resizing image: 50% complete... Resizing image: 51% complete... Resizing image: 53% complete... Resizing image: 54% complete... Resizing image: 56% complete... Resizing image: 57% complete... Resizing image: 59% complete... Resizing image: 60% complete... Resizing image: 62% complete... Resizing image: 64% complete... Resizing image: 65% complete... Resizing image: 67% complete... Resizing image: 68% complete... Resizing image: 70% complete... Resizing image: 71% complete... Resizing image: 73% complete... Resizing image: 75% complete... Resizing image: 76% complete... Resizing image: 78% complete... Resizing image: 79% complete... Resizing image: 81% complete... Resizing image: 82% complete... Resizing image: 84% complete... Resizing image: 85% complete... Resizing image: 87% complete... Resizing image: 89% complete... Resizing image: 90% complete... Resizing image: 92% complete... Resizing image: 93% complete... Resizing image: 95% complete... Resizing image: 96% complete... Resizing image: 98% complete... Resizing image: 100% complete...done. 2024-09-13T07:29:24.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img3 2024-09-13T07:29:24.931 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:24.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:29:24.938 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg1 2024-09-13T07:29:25.000 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:29:25.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg1 2024-09-13T07:29:25.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:25.066 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:25.067 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -rf /tmp/diff-testimg1-1 /tmp/diff-testimg1-2 2024-09-13T07:29:25.068 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export-diff testimg1 --snap=snap1 /tmp/diff-testimg1-1 2024-09-13T07:29:25.143 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 37% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:25.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export-diff testimg1 --from-snap=snap1 /tmp/diff-testimg1-2 2024-09-13T07:29:25.209 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:29:25.214 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --size=1 testimg-diff1 2024-09-13T07:29:25.294 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-1 testimg-diff1 2024-09-13T07:29:25.436 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image diff: 22% complete... Importing image diff: 63% complete... Importing image diff: 99% complete...2024-09-13T07:29:25.433+0000 7fe9b35c0700 0 -- 172.21.15.5:0/3812744040 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe994039cc0 msgr2=0x7fe99403c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:25.607 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:25.605+0000 7fe9b35c0700 0 -- 172.21.15.5:0/3812744040 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c4a601ea20 msgr2=0x7fe99403d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:25.612 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image diff: 100% complete...done. 2024-09-13T07:29:25.631 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-2 testimg-diff1 2024-09-13T07:29:25.743 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image diff: 68% complete... Importing image diff: 96% complete... Importing image diff: 100% complete...done. 2024-09-13T07:29:25.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg1 2024-09-13T07:29:25.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:29:25.827 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:29:25.828 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg1 2024-09-13T07:29:25.828 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:25.896 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:25.896 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff1 2024-09-13T07:29:25.896 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:29:25.963 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:29:25.964 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-09-13T07:29:25.964 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:26.031 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:26.032 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg1 --snap=snap1 testimg2 2024-09-13T07:29:26.196 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete...2024-09-13T07:29:26.193+0000 7f669d2a5700 0 -- 172.21.15.5:0/1460308065 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563951903d30 msgr2=0x5639519656e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:26.210 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:26.209+0000 7f669c043700 0 -- 172.21.15.5:0/1460308065 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f667c004b90 msgr2=0x7f667c005020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:26.213 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:29:26.218 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg1 testimg3 2024-09-13T07:29:26.563 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete...2024-09-13T07:29:26.561+0000 7f19b7a29700 0 -- 172.21.15.5:0/706199562 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f1990004be0 msgr2=0x7f1990005040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:26.579 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...2024-09-13T07:29:26.577+0000 7f19b8c8b700 0 -- 172.21.15.5:0/706199562 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562adeb206f0 msgr2=0x562adeb88280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:26.605 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:29:26.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg-diff1 --sparse-size 768K --snap=snap1 testimg-diff2 2024-09-13T07:29:26.751 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete...2024-09-13T07:29:26.749+0000 7f95e8184700 0 -- 172.21.15.5:0/4043003987 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562417b0c970 msgr2=0x562417b74bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:26.755 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:26.753+0000 7f95e8184700 0 -- 172.21.15.5:0/4043003987 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f95c8039cb0 msgr2=0x7f95c803c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:26.878 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:29:26.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg-diff1 --sparse-size 768K testimg-diff3 2024-09-13T07:29:27.021 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete...2024-09-13T07:29:27.017+0000 7ff998b74700 0 -- 172.21.15.5:0/3575491806 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7ff9700046e0 msgr2=0x7ff970004b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:27.028 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:27.025+0000 7ff998373700 0 -- 172.21.15.5:0/3575491806 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff9700046e0 msgr2=0x7ff97803ead0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:27.144 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-13T07:29:27.150 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg2 2024-09-13T07:29:27.150 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:27.206 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:27.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:29:27.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:29:27.268 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:29:27.268 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff2 2024-09-13T07:29:27.268 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:27.335 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:27.335 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff3 2024-09-13T07:29:27.335 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:29:27.405 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:29:27.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep copy testimg1 testimg4 2024-09-13T07:29:27.622 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:27.621+0000 7fe773d0c700 0 -- 172.21.15.5:0/588259170 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fe754004b90 msgr2=0x7fe754005020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:27.654 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete...2024-09-13T07:29:27.653+0000 7fe774f6e700 0 -- 172.21.15.5:0/588259170 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x563785b3c7e0 msgr2=0x563785ba3760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:27.654 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 100% complete...done. 2024-09-13T07:29:27.660 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep copy testimg1 --snap=snap1 testimg5 2024-09-13T07:29:28.638 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:28.637+0000 7f12116b4700 0 -- 172.21.15.5:0/2069847481 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x5584c834b270 msgr2=0x5584c834d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:28.664 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-09-13T07:29:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg4 2024-09-13T07:29:28.670 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:29:28.730 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:29:28.731 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg5 2024-09-13T07:29:28.732 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:28.796 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:28.797 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg4 2024-09-13T07:29:28.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:29:28.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:29:28.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:29:28.863 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:29:28.864 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg4 2024-09-13T07:29:28.864 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '.*snap1.*' 2024-09-13T07:29:28.926 INFO:tasks.workunit.client.0.smithi005.stdout: 20 snap1 256 MiB Fri Sep 13 07:29:27 2024 2024-09-13T07:29:28.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img1.new 2024-09-13T07:29:29.021 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:29.030 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg2 /tmp/img2.new 2024-09-13T07:29:29.140 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:29.147 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg3 /tmp/img3.new 2024-09-13T07:29:29.235 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:29.240 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.new 2024-09-13T07:29:29.335 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:29.342 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff2 /tmp/img-diff2.new 2024-09-13T07:29:29.464 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:29.471 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff3 /tmp/img-diff3.new 2024-09-13T07:29:29.559 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:29.564 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img2.new 2024-09-13T07:29:29.796 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img3 /tmp/img3.new 2024-09-13T07:29:29.914 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img-diff2.new 2024-09-13T07:29:30.110 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img3 /tmp/img-diff3.new 2024-09-13T07:29:30.210 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rollback --snap=snap1 testimg1 2024-09-13T07:29:30.332 INFO:tasks.workunit.client.0.smithi005.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-09-13T07:29:30.345 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rollback --snap=snap1 testimg-diff1 2024-09-13T07:29:30.469 INFO:tasks.workunit.client.0.smithi005.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-09-13T07:29:30.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg1 2024-09-13T07:29:30.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:30.544 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:30.544 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff1 2024-09-13T07:29:30.544 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:29:30.609 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:29:30.609 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img1.snap1 2024-09-13T07:29:30.736 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:30.743 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.snap1 2024-09-13T07:29:30.861 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:29:30.866 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img1.snap1 2024-09-13T07:29:31.055 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img-diff1.snap1 2024-09-13T07:29:31.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm testimg2 2024-09-13T07:29:31.415 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-09-13T07:29:31.413+0000 7ff9580f0700 0 -- 172.21.15.5:0/1243530311 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55b851e4c380 msgr2=0x55b851e4e810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:31.442 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:31.441+0000 7ff959352700 0 -- 172.21.15.5:0/1243530311 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b851d3fc00 msgr2=0x55b851dfe180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:31.444 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:31.449 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm testimg3 2024-09-13T07:29:31.622 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:29:31.621+0000 7ff4c3fc3700 0 -- 172.21.15.5:0/11548848 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564762e8d280 msgr2=0x564762ea5750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:31.648 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:31.645+0000 7ff4c2d61700 0 -- 172.21.15.5:0/11548848 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x564762ef3550 msgr2=0x564762ef59e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:31.656 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:31.662 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create testimg2 -s 0 2024-09-13T07:29:31.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd cp testimg2 testimg3 2024-09-13T07:29:31.861 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:29:31.868 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep cp testimg2 testimg6 2024-09-13T07:29:32.019 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 100% complete...done. 2024-09-13T07:29:32.025 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm --snap=snap1 testimg1 2024-09-13T07:29:32.628 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:29:32.641 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm --snap=snap1 testimg-diff1 2024-09-13T07:29:33.636 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:29:33.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg1 2024-09-13T07:29:33.654 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-09-13T07:29:33.717 INFO:tasks.workunit.client.0.smithi005.stdout:error setting snapshot context: (2) No such file or directory 2024-09-13T07:29:33.718 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-09-13T07:29:33.718 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-09-13T07:29:33.784 INFO:tasks.workunit.client.0.smithi005.stdout:error setting snapshot context: (2) No such file or directory 2024-09-13T07:29:33.785 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd sparsify testimg1 2024-09-13T07:29:33.899 INFO:tasks.workunit.client.0.smithi005.stderr: Image sparsify: 1% complete... Image sparsify: 3% complete... Image sparsify: 4% complete... Image sparsify: 6% complete... Image sparsify: 7% complete... Image sparsify: 9% complete... Image sparsify: 10% complete... Image sparsify: 12% complete... Image sparsify: 14% complete... Image sparsify: 15% complete... Image sparsify: 17% complete... Image sparsify: 18% complete... Image sparsify: 20% complete... Image sparsify: 21% complete... Image sparsify: 23% complete... Image sparsify: 25% complete... Image sparsify: 26% complete... Image sparsify: 28% complete... Image sparsify: 29% complete... Image sparsify: 31% complete... Image sparsify: 32% complete... Image sparsify: 34% complete... Image sparsify: 35% complete... Image sparsify: 37% complete... Image sparsify: 39% complete... Image sparsify: 40% complete... Image sparsify: 42% complete... Image sparsify: 43% complete... Image sparsify: 45% complete... Image sparsify: 46% complete... Image sparsify: 48% complete... Image sparsify: 50% complete... Image sparsify: 51% complete... Image sparsify: 53% complete... Image sparsify: 54% complete... Image sparsify: 56% complete... Image sparsify: 57% complete... Image sparsify: 59% complete... Image sparsify: 60% complete... Image sparsify: 62% complete... Image sparsify: 64% complete... Image sparsify: 65% complete... Image sparsify: 67% complete... Image sparsify: 68% complete... Image sparsify: 70% complete... Image sparsify: 71% complete... Image sparsify: 73% complete... Image sparsify: 75% complete... Image sparsify: 76% complete... Image sparsify: 78% complete... Image sparsify: 79% complete... Image sparsify: 81% complete... Image sparsify: 82% complete... Image sparsify: 84% complete... Image sparsify: 85% complete... Image sparsify: 87% complete... Image sparsify: 89% complete... Image sparsify: 90% complete... Image sparsify: 92% complete... Image sparsify: 93% complete... Image sparsify: 95% complete... Image sparsify: 96% complete... Image sparsify: 98% complete... Image sparsify: 100% complete...done. 2024-09-13T07:29:33.915 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:33.915 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:34.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:34.505 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:34.790 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:35.896 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:36.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:37.196 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:37.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:37.770 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.347 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.491 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.637 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.789 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:38.932 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:39.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:39.228 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:39.371 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-09-13T07:29:39.475 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_locking 2024-09-13T07:29:39.475 INFO:tasks.workunit.client.0.smithi005.stdout:testing locking... 2024-09-13T07:29:39.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing locking...' 2024-09-13T07:29:39.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:39.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:39.608 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:39.755 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:39.908 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.057 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.206 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.347 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.491 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.637 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.780 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:40.927 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:41.486 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:41.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:41.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:42.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:42.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:42.377 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:42.522 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create -s 1 test1 2024-09-13T07:29:42.615 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:42.615 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:29:42.616 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:29:42.682 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:29:42.682 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id 2024-09-13T07:29:42.758 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:42.758 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 1 ' 2024-09-13T07:29:42.824 INFO:tasks.workunit.client.0.smithi005.stdout:There is 1 exclusive lock on this image. 2024-09-13T07:29:42.825 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:29:42.825 INFO:tasks.workunit.client.0.smithi005.stderr:++ tail -n 1 2024-09-13T07:29:42.825 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{print $1;}' 2024-09-13T07:29:42.893 INFO:tasks.workunit.client.0.smithi005.stderr:+ LOCKER=client.7822 2024-09-13T07:29:42.893 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock remove test1 id client.7822 2024-09-13T07:29:43.665 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:43.665 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:29:43.665 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:29:43.730 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:29:43.731 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id --shared tag 2024-09-13T07:29:43.812 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:43.812 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 1 ' 2024-09-13T07:29:43.878 INFO:tasks.workunit.client.0.smithi005.stdout:There is 1 shared lock on this image. 2024-09-13T07:29:43.878 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id --shared tag 2024-09-13T07:29:43.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:43.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 2 ' 2024-09-13T07:29:44.023 INFO:tasks.workunit.client.0.smithi005.stdout:There are 2 shared locks on this image. 2024-09-13T07:29:44.023 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id2 --shared tag 2024-09-13T07:29:44.106 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:44.106 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 3 ' 2024-09-13T07:29:44.172 INFO:tasks.workunit.client.0.smithi005.stdout:There are 3 shared locks on this image. 2024-09-13T07:29:44.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:44.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ tail -n 1 2024-09-13T07:29:44.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ awk '{print $2, $1;}' 2024-09-13T07:29:44.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ xargs rbd lock remove test1 2024-09-13T07:29:44.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:29:44.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -qE 'features:.*exclusive' 2024-09-13T07:29:44.739 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:29:44.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' -n 'There are 2 shared locks on this image. 2024-09-13T07:29:44.804 INFO:tasks.workunit.client.0.smithi005.stderr:Lock tag: tag 2024-09-13T07:29:44.804 INFO:tasks.workunit.client.0.smithi005.stderr:Locker ID Address 2024-09-13T07:29:44.804 INFO:tasks.workunit.client.0.smithi005.stderr:client.7837 id 172.21.15.5:0/345399310 2024-09-13T07:29:44.804 INFO:tasks.workunit.client.0.smithi005.stderr:client.7843 id 172.21.15.5:0/3035598857' ']' 2024-09-13T07:29:44.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:44.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ tail -n 1 2024-09-13T07:29:44.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ awk '{print $2, $1;}' 2024-09-13T07:29:44.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ xargs rbd lock remove test1 2024-09-13T07:29:45.655 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:29:45.712 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' -n 'There is 1 shared lock on this image. 2024-09-13T07:29:45.713 INFO:tasks.workunit.client.0.smithi005.stderr:Lock tag: tag 2024-09-13T07:29:45.713 INFO:tasks.workunit.client.0.smithi005.stderr:Locker ID Address 2024-09-13T07:29:45.713 INFO:tasks.workunit.client.0.smithi005.stderr:client.7837 id 172.21.15.5:0/345399310' ']' 2024-09-13T07:29:45.713 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:29:45.713 INFO:tasks.workunit.client.0.smithi005.stderr:+ tail -n 1 2024-09-13T07:29:45.714 INFO:tasks.workunit.client.0.smithi005.stderr:+ awk '{print $2, $1;}' 2024-09-13T07:29:45.714 INFO:tasks.workunit.client.0.smithi005.stderr:+ xargs rbd lock remove test1 2024-09-13T07:29:46.676 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:29:46.742 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' -n '' ']' 2024-09-13T07:29:46.742 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:29:47.142 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:47.141+0000 7fce5c3a3700 0 -- 172.21.15.5:0/1625794547 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x56548705b630 msgr2=0x56548705da20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:47.174 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:47.179 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_thick_provision 2024-09-13T07:29:47.180 INFO:tasks.workunit.client.0.smithi005.stdout:testing thick provision... 2024-09-13T07:29:47.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing thick provision...' 2024-09-13T07:29:47.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:47.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:47.328 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:47.475 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:47.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:47.766 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:47.912 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:48.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:48.214 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:48.360 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:48.713 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:48.856 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.007 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.158 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.308 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.752 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:49.902 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:50.050 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --thick-provision -s 64M test1 2024-09-13T07:29:50.239 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 6% complete... Thick provisioning: 12% complete... Thick provisioning: 18% complete... Thick provisioning: 25% complete... Thick provisioning: 31% complete... Thick provisioning: 37% complete... Thick provisioning: 43% complete... Thick provisioning: 50% complete... Thick provisioning: 56% complete... Thick provisioning: 62% complete... Thick provisioning: 68% complete... Thick provisioning: 75% complete... Thick provisioning: 81% complete... Thick provisioning: 87% complete... Thick provisioning: 93% complete... Thick provisioning: 100% complete...2024-09-13T07:29:50.237+0000 7f8fa96f4700 0 -- 172.21.15.5:0/2547379834 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f8f880055c0 msgr2=0x7f8f88006f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:50.253 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 100% complete...done. 2024-09-13T07:29:50.263 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:50.261+0000 7f8fab157700 0 -- 172.21.15.5:0/2547379834 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564b07d255a0 msgr2=0x564b07d3b6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:50.270 INFO:tasks.workunit.client.0.smithi005.stderr:+ count=0 2024-09-13T07:29:50.270 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret= 2024-09-13T07:29:50.270 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 -lt 10 ']' 2024-09-13T07:29:50.270 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:29:50.270 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:29:50.271 INFO:tasks.workunit.client.0.smithi005.stderr:+ tr -s ' ' 2024-09-13T07:29:50.271 INFO:tasks.workunit.client.0.smithi005.stderr:+ cut -d ' ' -f 4-5 2024-09-13T07:29:50.271 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^64 MiB' 2024-09-13T07:29:50.332 INFO:tasks.workunit.client.0.smithi005.stdout:64 MiB 2024-09-13T07:29:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret=0 2024-09-13T07:29:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 = 0 ']' 2024-09-13T07:29:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:29:50.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:29:50.391 INFO:tasks.workunit.client.0.smithi005.stdout:NAME PROVISIONED USED 2024-09-13T07:29:50.391 INFO:tasks.workunit.client.0.smithi005.stdout:test1 64 MiB 64 MiB 2024-09-13T07:29:50.396 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 '!=' 0 ']' 2024-09-13T07:29:50.397 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:29:50.578 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-13T07:29:50.577+0000 7fb9dfaab700 0 -- 172.21.15.5:0/2567699565 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55f066d4f100 msgr2=0x55f066d66e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:50.599 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:50.597+0000 7fb9dfaab700 0 -- 172.21.15.5:0/2567699565 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fb9c4039db0 msgr2=0x7fb9c403c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:50.621 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:29:50.627 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:29:50.627 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:29:50.628 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:29:50.628 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:29:50.678 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:29:50.678 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --thick-provision -s 4G test1 2024-09-13T07:29:50.861 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 1% complete... Thick provisioning: 2% complete...2024-09-13T07:29:50.857+0000 7ff2163f0700 0 -- 172.21.15.5:0/1773011241 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7ff1f8039df0 msgr2=0x7ff1f803c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:50.893 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:29:50.889+0000 7ff2163f0700 0 -- 172.21.15.5:0/1773011241 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x559b1b906e60 msgr2=0x7ff1f803ce10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:52.395 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:29:52.393+0000 7f3284771700 -1 reset not still connected to 0x5597c427e340 2024-09-13T07:29:56.561 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:29:56.557+0000 7f3284771700 -1 reset not still connected to 0x5597c427e340 2024-09-13T07:29:56.709 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 3% complete... Thick provisioning: 4% complete... Thick provisioning: 5% complete... Thick provisioning: 6% complete... Thick provisioning: 7% complete... Thick provisioning: 8% complete... Thick provisioning: 9% complete... Thick provisioning: 10% complete... Thick provisioning: 11% complete... Thick provisioning: 12% complete... Thick provisioning: 13% complete... Thick provisioning: 14% complete... Thick provisioning: 15% complete... Thick provisioning: 16% complete... Thick provisioning: 17% complete... Thick provisioning: 18% complete... Thick provisioning: 19% complete... Thick provisioning: 20% complete... Thick provisioning: 21% complete... Thick provisioning: 22% complete... Thick provisioning: 23% complete... Thick provisioning: 24% complete... Thick provisioning: 25% complete... Thick provisioning: 26% complete... Thick provisioning: 27% complete... Thick provisioning: 28% complete... Thick provisioning: 29% complete... Thick provisioning: 30% complete... Thick provisioning: 31% complete... Thick provisioning: 32% complete... Thick provisioning: 33% complete... Thick provisioning: 34% complete... Thick provisioning: 35% complete... Thick provisioning: 36% complete... Thick provisioning: 37% complete... Thick provisioning: 38% complete... Thick provisioning: 39% complete... Thick provisioning: 40% complete... Thick provisioning: 41% complete... Thick provisioning: 42% complete... Thick provisioning: 43% complete... Thick provisioning: 44% complete... Thick provisioning: 45% complete... Thick provisioning: 46% complete... Thick provisioning: 47% complete... Thick provisioning: 48% complete... Thick provisioning: 49% complete... Thick provisioning: 50% complete... Thick provisioning: 51% complete... Thick provisioning: 52% complete... Thick provisioning: 53% complete... Thick provisioning: 54% complete... Thick provisioning: 55% complete... Thick provisioning: 56% complete... Thick provisioning: 57% complete... Thick provisioning: 58% complete... Thick provisioning: 59% complete... Thick provisioning: 60% complete... Thick provisioning: 61% complete... Thick provisioning: 62% complete... Thick provisioning: 63% complete... Thick provisioning: 64% complete... Thick provisioning: 65% complete... Thick provisioning: 66% complete... Thick provisioning: 67% complete... Thick provisioning: 68% complete... Thick provisioning: 69% complete... Thick provisioning: 70% complete... Thick provisioning: 71% complete... Thick provisioning: 72% complete... Thick provisioning: 73% complete... Thick provisioning: 74% complete... Thick provisioning: 75% complete... Thick provisioning: 76% complete... Thick provisioning: 77% complete... Thick provisioning: 78% complete... Thick provisioning: 79% complete... Thick provisioning: 80% complete... Thick provisioning: 81% complete... Thick provisioning: 82% complete... Thick provisioning: 83% complete... Thick provisioning: 84% complete... Thick provisioning: 85% complete... Thick provisioning: 86% complete... Thick provisioning: 87% complete... Thick provisioning: 88% complete... Thick provisioning: 89% complete... Thick provisioning: 90% complete... Thick provisioning: 91% complete... Thick provisioning: 92% complete... Thick provisioning: 93% complete... Thick provisioning: 94% complete... Thick provisioning: 95% complete... Thick provisioning: 96% complete... Thick provisioning: 97% complete... Thick provisioning: 98% complete... Thick provisioning: 99% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2024-09-13T07:29:56.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ count=0 2024-09-13T07:29:56.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret= 2024-09-13T07:29:56.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 -lt 10 ']' 2024-09-13T07:29:56.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:29:56.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:29:56.728 INFO:tasks.workunit.client.0.smithi005.stderr:+ tr -s ' ' 2024-09-13T07:29:56.728 INFO:tasks.workunit.client.0.smithi005.stderr:+ cut -d ' ' -f 4-5 2024-09-13T07:29:56.728 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^4 GiB' 2024-09-13T07:29:56.790 INFO:tasks.workunit.client.0.smithi005.stdout:4 GiB 2024-09-13T07:29:56.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret=0 2024-09-13T07:29:56.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 = 0 ']' 2024-09-13T07:29:56.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:29:56.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:29:56.845 INFO:tasks.workunit.client.0.smithi005.stdout:NAME PROVISIONED USED 2024-09-13T07:29:56.845 INFO:tasks.workunit.client.0.smithi005.stdout:test1 4 GiB 4 GiB 2024-09-13T07:29:56.850 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 '!=' 0 ']' 2024-09-13T07:29:56.850 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:29:57.040 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete...2024-09-13T07:29:57.037+0000 7fba65458700 0 -- 172.21.15.5:0/4026789752 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55eb7ade2830 msgr2=0x55eb7ade4c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:29:57.055 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 4% complete...2024-09-13T07:29:57.053+0000 7fba641f6700 0 -- 172.21.15.5:0/4026789752 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55eb7ae32b70 msgr2=0x55eb7ae35000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:29:57.861 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T07:29:57.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:29:57.866 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:29:57.866 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:29:57.866 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:29:57.908 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:29:57.909 INFO:tasks.workunit.client.0.smithi005.stderr:+ RBD_CREATE_ARGS='--image-format 2' 2024-09-13T07:29:57.909 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_others 2024-09-13T07:29:57.909 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing import, export, resize, and snapshots...' 2024-09-13T07:29:57.909 INFO:tasks.workunit.client.0.smithi005.stdout:testing import, export, resize, and snapshots... 2024-09-13T07:29:57.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ TMP_FILES='/tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1' 2024-09-13T07:29:57.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:29:57.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:58.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:58.179 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:58.316 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:58.461 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:58.605 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:58.751 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.111 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.554 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.700 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.844 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:29:59.995 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:00.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:00.294 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:00.445 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:00.600 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:00.741 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-09-13T07:30:00.743 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/sh of=/tmp/img1 bs=1k count=1 seek=10 2024-09-13T07:30:00.744 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records in 2024-09-13T07:30:00.744 INFO:tasks.workunit.client.0.smithi005.stderr:1+0 records out 2024-09-13T07:30:00.744 INFO:tasks.workunit.client.0.smithi005.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000157029 s, 6.5 MB/s 2024-09-13T07:30:00.744 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/dd of=/tmp/img1 bs=1k count=10 seek=100 2024-09-13T07:30:00.746 INFO:tasks.workunit.client.0.smithi005.stderr:10+0 records in 2024-09-13T07:30:00.746 INFO:tasks.workunit.client.0.smithi005.stderr:10+0 records out 2024-09-13T07:30:00.746 INFO:tasks.workunit.client.0.smithi005.stderr:10240 bytes (10 kB, 10 KiB) copied, 0.000222435 s, 46.0 MB/s 2024-09-13T07:30:00.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/rm of=/tmp/img1 bs=1k count=100 seek=1000 2024-09-13T07:30:00.748 INFO:tasks.workunit.client.0.smithi005.stderr:70+1 records in 2024-09-13T07:30:00.748 INFO:tasks.workunit.client.0.smithi005.stderr:70+1 records out 2024-09-13T07:30:00.748 INFO:tasks.workunit.client.0.smithi005.stderr:72056 bytes (72 kB, 70 KiB) copied, 0.000479128 s, 150 MB/s 2024-09-13T07:30:00.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/ls of=/tmp/img1 bs=1k seek=10000 2024-09-13T07:30:00.750 INFO:tasks.workunit.client.0.smithi005.stderr:138+1 records in 2024-09-13T07:30:00.750 INFO:tasks.workunit.client.0.smithi005.stderr:138+1 records out 2024-09-13T07:30:00.750 INFO:tasks.workunit.client.0.smithi005.stderr:142144 bytes (142 kB, 139 KiB) copied, 0.000851222 s, 167 MB/s 2024-09-13T07:30:00.750 INFO:tasks.workunit.client.0.smithi005.stderr:+ dd if=/bin/ln of=/tmp/img1 bs=1k seek=100000 2024-09-13T07:30:00.752 INFO:tasks.workunit.client.0.smithi005.stderr:74+1 records in 2024-09-13T07:30:00.752 INFO:tasks.workunit.client.0.smithi005.stderr:74+1 records out 2024-09-13T07:30:00.752 INFO:tasks.workunit.client.0.smithi005.stderr:76160 bytes (76 kB, 74 KiB) copied, 0.000605091 s, 126 MB/s 2024-09-13T07:30:00.752 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import --image-format 2 /tmp/img1 testimg1 2024-09-13T07:30:00.953 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 4% complete... Importing image: 8% complete... Importing image: 12% complete... Importing image: 16% complete... Importing image: 20% complete... Importing image: 24% complete... Importing image: 28% complete... Importing image: 32% complete... Importing image: 36% complete... Importing image: 40% complete... Importing image: 45% complete... Importing image: 49% complete... Importing image: 53% complete... Importing image: 57% complete... Importing image: 61% complete... Importing image: 65% complete... Importing image: 69% complete... Importing image: 73% complete... Importing image: 77% complete... Importing image: 81% complete... Importing image: 85% complete... Importing image: 90% complete... Importing image: 94% complete... Importing image: 98% complete...2024-09-13T07:30:00.949+0000 7f65253af700 0 -- 172.21.15.5:0/3090749868 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f650c0041f0 msgr2=0x7f650c006640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:30:00.957 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image: 100% complete...done. 2024-09-13T07:30:00.963 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd resize testimg1 --size=256 --allow-shrink 2024-09-13T07:30:01.059 INFO:tasks.workunit.client.0.smithi005.stderr: Resizing image: 100% complete...done. 2024-09-13T07:30:01.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img2 2024-09-13T07:30:01.208 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:01.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1 --snap=snap1 2024-09-13T07:30:01.704 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:01.720 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd resize testimg1 --size=128 2024-09-13T07:30:01.785 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: shrinking an image is only allowed with the --allow-shrink flag 2024-09-13T07:30:01.794 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:30:01.794 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd resize testimg1 --size=128 --allow-shrink 2024-09-13T07:30:01.907 INFO:tasks.workunit.client.0.smithi005.stderr: Resizing image: 50% complete... Resizing image: 51% complete... Resizing image: 53% complete... Resizing image: 54% complete... Resizing image: 56% complete... Resizing image: 57% complete... Resizing image: 59% complete... Resizing image: 60% complete... Resizing image: 62% complete... Resizing image: 64% complete... Resizing image: 65% complete... Resizing image: 67% complete... Resizing image: 68% complete... Resizing image: 70% complete... Resizing image: 71% complete... Resizing image: 73% complete... Resizing image: 75% complete... Resizing image: 76% complete... Resizing image: 78% complete... Resizing image: 79% complete... Resizing image: 81% complete... Resizing image: 82% complete... Resizing image: 84% complete... Resizing image: 85% complete... Resizing image: 87% complete... Resizing image: 89% complete... Resizing image: 90% complete... Resizing image: 92% complete... Resizing image: 93% complete... Resizing image: 95% complete... Resizing image: 96% complete... Resizing image: 98% complete... Resizing image: 100% complete...done. 2024-09-13T07:30:01.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img3 2024-09-13T07:30:02.025 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:02.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg1 2024-09-13T07:30:02.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:30:02.095 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:30:02.096 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg1 2024-09-13T07:30:02.096 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:02.159 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:02.160 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -rf /tmp/diff-testimg1-1 /tmp/diff-testimg1-2 2024-09-13T07:30:02.162 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export-diff testimg1 --snap=snap1 /tmp/diff-testimg1-1 2024-09-13T07:30:02.240 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 37% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:02.244 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export-diff testimg1 --from-snap=snap1 /tmp/diff-testimg1-2 2024-09-13T07:30:02.311 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:30:02.318 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size=1 testimg-diff1 2024-09-13T07:30:02.412 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-1 testimg-diff1 2024-09-13T07:30:02.563 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image diff: 22% complete... Importing image diff: 63% complete... Importing image diff: 99% complete...2024-09-13T07:30:02.561+0000 7fa7a02c9700 0 -- 172.21.15.5:0/2330528780 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5573f3a11a10 msgr2=0x5573f3a76a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:02.701 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:02.697+0000 7fa7a02c9700 0 -- 172.21.15.5:0/2330528780 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fa784039df0 msgr2=0x7fa78403c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:02.706 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image diff: 100% complete...done. 2024-09-13T07:30:02.725 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-2 testimg-diff1 2024-09-13T07:30:02.830 INFO:tasks.workunit.client.0.smithi005.stderr: Importing image diff: 68% complete... Importing image diff: 96% complete... Importing image diff: 100% complete...done. 2024-09-13T07:30:02.848 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg1 2024-09-13T07:30:02.848 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:30:02.909 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:30:02.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg1 2024-09-13T07:30:02.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:02.985 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:02.986 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff1 2024-09-13T07:30:02.986 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:30:03.054 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:30:03.054 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-09-13T07:30:03.054 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:03.127 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:03.127 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg1 --snap=snap1 testimg2 2024-09-13T07:30:03.302 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete...2024-09-13T07:30:03.301+0000 7f2703791700 0 -- 172.21.15.5:0/191083871 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x560b0739fae0 msgr2=0x560b07404780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:03.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:03.317+0000 7f2703791700 0 -- 172.21.15.5:0/191083871 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f26e4039e00 msgr2=0x7f26e403c210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:03.319 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:30:03.325 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg1 testimg3 2024-09-13T07:30:03.467 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete...2024-09-13T07:30:03.465+0000 7fe9b08b2700 0 -- 172.21.15.5:0/68387188 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fe990004bb0 msgr2=0x7fe990007640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:30:03.483 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:03.481+0000 7fe9b08b2700 0 -- 172.21.15.5:0/68387188 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fe994039ed0 msgr2=0x7fe99403c2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:03.509 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:30:03.515 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg-diff1 --sparse-size 768K --snap=snap1 testimg-diff2 2024-09-13T07:30:03.665 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 37% complete...2024-09-13T07:30:03.661+0000 7fb97fc23700 0 -- 172.21.15.5:0/2970203438 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564138f5a0d0 msgr2=0x564138fbc530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:30:03.677 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...2024-09-13T07:30:03.673+0000 7fb97fc23700 0 -- 172.21.15.5:0/2970203438 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fb960039ec0 msgr2=0x7fb96003c2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:30:03.815 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:30:03.821 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd copy testimg-diff1 --sparse-size 768K testimg-diff3 2024-09-13T07:30:03.967 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete...2024-09-13T07:30:03.965+0000 7f7adce2e700 0 -- 172.21.15.5:0/923853690 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x5626962a3dc0 msgr2=0x5626962a6260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:30:03.979 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:03.977+0000 7f7ade891700 0 -- 172.21.15.5:0/923853690 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562696240b80 msgr2=0x5626962a2880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:30:04.106 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2024-09-13T07:30:04.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg2 2024-09-13T07:30:04.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:04.166 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:04.166 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:30:04.166 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:30:04.224 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:30:04.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff2 2024-09-13T07:30:04.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:04.282 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:04.282 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff3 2024-09-13T07:30:04.282 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:30:04.348 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:30:04.349 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep copy testimg1 testimg4 2024-09-13T07:30:04.714 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:04.713+0000 7f61dbf34700 0 -- 172.21.15.5:0/1626480423 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x558572dc7d80 msgr2=0x558572e29740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:04.749 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete...2024-09-13T07:30:04.749+0000 7f61dbf34700 0 -- 172.21.15.5:0/1626480423 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f61c003a310 msgr2=0x7f61c003c700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:04.755 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 100% complete...done. 2024-09-13T07:30:04.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep copy testimg1 --snap=snap1 testimg5 2024-09-13T07:30:05.729 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:05.725+0000 7fcfc4d37700 0 -- 172.21.15.5:0/149614129 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b1b656f900 msgr2=0x55b1b65d4400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:05.755 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-09-13T07:30:05.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg4 2024-09-13T07:30:05.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 128 MiB' 2024-09-13T07:30:05.828 INFO:tasks.workunit.client.0.smithi005.stdout: size 128 MiB in 32 objects 2024-09-13T07:30:05.829 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg5 2024-09-13T07:30:05.829 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:05.893 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:05.894 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg4 2024-09-13T07:30:05.894 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:30:05.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:05.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:05.959 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg4 2024-09-13T07:30:05.960 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '.*snap1.*' 2024-09-13T07:30:06.027 INFO:tasks.workunit.client.0.smithi005.stdout: 28 snap1 256 MiB Fri Sep 13 07:30:04 2024 2024-09-13T07:30:06.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img1.new 2024-09-13T07:30:06.141 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:06.147 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg2 /tmp/img2.new 2024-09-13T07:30:06.264 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:06.269 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg3 /tmp/img3.new 2024-09-13T07:30:06.355 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:06.360 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.new 2024-09-13T07:30:06.454 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:06.459 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff2 /tmp/img-diff2.new 2024-09-13T07:30:06.578 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:06.583 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff3 /tmp/img-diff3.new 2024-09-13T07:30:06.676 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:06.683 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img2.new 2024-09-13T07:30:06.913 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img3 /tmp/img3.new 2024-09-13T07:30:07.026 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img-diff2.new 2024-09-13T07:30:07.223 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img3 /tmp/img-diff3.new 2024-09-13T07:30:07.323 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rollback --snap=snap1 testimg1 2024-09-13T07:30:07.438 INFO:tasks.workunit.client.0.smithi005.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-09-13T07:30:07.444 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:07.440+0000 7f86dcf52700 0 -- 172.21.15.5:0/1919091378 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f86b8008be0 msgr2=0x7f86b800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:07.451 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rollback --snap=snap1 testimg-diff1 2024-09-13T07:30:07.571 INFO:tasks.workunit.client.0.smithi005.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2024-09-13T07:30:07.578 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:07.576+0000 7f1f86a2e700 0 -- 172.21.15.5:0/3732092027 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5639b5885830 msgr2=0x5639b5887c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:07.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg1 2024-09-13T07:30:07.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:07.652 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:07.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg-diff1 2024-09-13T07:30:07.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:30:07.717 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:30:07.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg1 /tmp/img1.snap1 2024-09-13T07:30:07.843 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:07.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.snap1 2024-09-13T07:30:07.964 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2024-09-13T07:30:07.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img1.snap1 2024-09-13T07:30:08.160 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /tmp/img2 /tmp/img-diff1.snap1 2024-09-13T07:30:08.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm testimg2 2024-09-13T07:30:08.537 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-09-13T07:30:08.532+0000 7f5f6ebe3700 0 -- 172.21.15.5:0/3099114875 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55d31c5dc0d0 msgr2=0x55d31c5f42c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:08.570 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:08.568+0000 7f5f6ebe3700 0 -- 172.21.15.5:0/3099114875 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f5f50039ed0 msgr2=0x7f5f5003c2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:08.571 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:08.577 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm testimg3 2024-09-13T07:30:08.753 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2024-09-13T07:30:08.748+0000 7fabde189700 0 -- 172.21.15.5:0/905920463 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562216e39100 msgr2=0x562216e51310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:08.785 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:08.780+0000 7fabde189700 0 -- 172.21.15.5:0/905920463 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fabc0039db0 msgr2=0x7fabc003c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:08.800 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:08.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create testimg2 -s 0 2024-09-13T07:30:08.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd cp testimg2 testimg3 2024-09-13T07:30:09.012 INFO:tasks.workunit.client.0.smithi005.stderr: Image copy: 100% complete...done. 2024-09-13T07:30:09.018 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep cp testimg2 testimg6 2024-09-13T07:30:09.369 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 100% complete...done. 2024-09-13T07:30:09.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm --snap=snap1 testimg1 2024-09-13T07:30:10.323 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:30:10.337 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm --snap=snap1 testimg-diff1 2024-09-13T07:30:11.331 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:30:11.346 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg1 2024-09-13T07:30:11.346 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-09-13T07:30:11.408 INFO:tasks.workunit.client.0.smithi005.stdout:error setting snapshot context: (2) No such file or directory 2024-09-13T07:30:11.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --snap=snap1 testimg-diff1 2024-09-13T07:30:11.409 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2024-09-13T07:30:11.468 INFO:tasks.workunit.client.0.smithi005.stdout:error setting snapshot context: (2) No such file or directory 2024-09-13T07:30:11.468 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd sparsify testimg1 2024-09-13T07:30:11.581 INFO:tasks.workunit.client.0.smithi005.stderr: Image sparsify: 1% complete... Image sparsify: 3% complete... Image sparsify: 4% complete... Image sparsify: 6% complete... Image sparsify: 7% complete... Image sparsify: 9% complete... Image sparsify: 10% complete... Image sparsify: 12% complete... Image sparsify: 14% complete... Image sparsify: 15% complete... Image sparsify: 17% complete... Image sparsify: 18% complete... Image sparsify: 20% complete... Image sparsify: 21% complete... Image sparsify: 23% complete... Image sparsify: 25% complete... Image sparsify: 26% complete... Image sparsify: 28% complete... Image sparsify: 29% complete... Image sparsify: 31% complete... Image sparsify: 32% complete... Image sparsify: 34% complete... Image sparsify: 35% complete... Image sparsify: 37% complete... Image sparsify: 39% complete... Image sparsify: 40% complete... Image sparsify: 42% complete... Image sparsify: 43% complete... Image sparsify: 45% complete... Image sparsify: 46% complete... Image sparsify: 48% complete... Image sparsify: 50% complete... Image sparsify: 51% complete... Image sparsify: 53% complete... Image sparsify: 54% complete... Image sparsify: 56% complete... Image sparsify: 57% complete... Image sparsify: 59% complete... Image sparsify: 60% complete... Image sparsify: 62% complete... Image sparsify: 64% complete... Image sparsify: 65% complete... Image sparsify: 67% complete... Image sparsify: 68% complete... Image sparsify: 70% complete... Image sparsify: 71% complete... Image sparsify: 73% complete... Image sparsify: 75% complete... Image sparsify: 76% complete... Image sparsify: 78% complete... Image sparsify: 79% complete... Image sparsify: 81% complete... Image sparsify: 82% complete... Image sparsify: 84% complete... Image sparsify: 85% complete... Image sparsify: 87% complete... Image sparsify: 89% complete... Image sparsify: 90% complete... Image sparsify: 92% complete... Image sparsify: 93% complete... Image sparsify: 95% complete... Image sparsify: 96% complete... Image sparsify: 98% complete... Image sparsify: 100% complete...done. 2024-09-13T07:30:11.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:30:11.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:11.891 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:12.187 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:12.465 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:13.582 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:14.606 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:14.903 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:15.193 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:15.541 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:15.840 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:15.987 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:16.133 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:16.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:16.427 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:16.568 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:16.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:17.077 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:17.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:17.374 INFO:tasks.workunit.client.0.smithi005.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2024-09-13T07:30:17.479 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_locking 2024-09-13T07:30:17.479 INFO:tasks.workunit.client.0.smithi005.stdout:testing locking... 2024-09-13T07:30:17.479 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing locking...' 2024-09-13T07:30:17.480 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:30:17.480 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:17.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:17.759 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:17.907 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.054 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.210 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.359 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.509 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.657 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:18.956 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:19.104 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:19.250 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:19.607 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:19.753 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:19.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:20.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:20.196 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:20.350 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:30:20.443 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:20.444 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:20.444 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:30:20.511 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:20.511 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id 2024-09-13T07:30:20.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:20.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 1 ' 2024-09-13T07:30:20.650 INFO:tasks.workunit.client.0.smithi005.stdout:There is 1 exclusive lock on this image. 2024-09-13T07:30:20.651 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:30:20.651 INFO:tasks.workunit.client.0.smithi005.stderr:++ tail -n 1 2024-09-13T07:30:20.651 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{print $1;}' 2024-09-13T07:30:20.723 INFO:tasks.workunit.client.0.smithi005.stderr:+ LOCKER=client.8513 2024-09-13T07:30:20.723 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock remove test1 id client.8513 2024-09-13T07:30:21.670 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:21.671 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:21.671 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:30:21.730 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:21.730 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id --shared tag 2024-09-13T07:30:21.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:21.810 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 1 ' 2024-09-13T07:30:21.876 INFO:tasks.workunit.client.0.smithi005.stdout:There is 1 shared lock on this image. 2024-09-13T07:30:21.876 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id --shared tag 2024-09-13T07:30:21.961 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:21.961 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 2 ' 2024-09-13T07:30:22.030 INFO:tasks.workunit.client.0.smithi005.stdout:There are 2 shared locks on this image. 2024-09-13T07:30:22.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock add test1 id2 --shared tag 2024-09-13T07:30:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:22.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep ' 3 ' 2024-09-13T07:30:22.177 INFO:tasks.workunit.client.0.smithi005.stdout:There are 3 shared locks on this image. 2024-09-13T07:30:22.177 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:22.178 INFO:tasks.workunit.client.0.smithi005.stderr:+ tail -n 1 2024-09-13T07:30:22.178 INFO:tasks.workunit.client.0.smithi005.stderr:+ awk '{print $2, $1;}' 2024-09-13T07:30:22.178 INFO:tasks.workunit.client.0.smithi005.stderr:+ xargs rbd lock remove test1 2024-09-13T07:30:22.675 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info test1 2024-09-13T07:30:22.675 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -qE 'features:.*exclusive' 2024-09-13T07:30:22.739 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:30:22.803 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' -n 'There are 2 shared locks on this image. 2024-09-13T07:30:22.804 INFO:tasks.workunit.client.0.smithi005.stderr:Lock tag: tag 2024-09-13T07:30:22.804 INFO:tasks.workunit.client.0.smithi005.stderr:Locker ID Address 2024-09-13T07:30:22.804 INFO:tasks.workunit.client.0.smithi005.stderr:client.8528 id 172.21.15.5:0/3975735010 2024-09-13T07:30:22.804 INFO:tasks.workunit.client.0.smithi005.stderr:client.8534 id 172.21.15.5:0/3226133755' ']' 2024-09-13T07:30:22.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:22.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ tail -n 1 2024-09-13T07:30:22.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ awk '{print $2, $1;}' 2024-09-13T07:30:22.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ xargs rbd lock remove test1 2024-09-13T07:30:23.679 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:30:23.742 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' -n 'There is 1 shared lock on this image. 2024-09-13T07:30:23.742 INFO:tasks.workunit.client.0.smithi005.stderr:Lock tag: tag 2024-09-13T07:30:23.742 INFO:tasks.workunit.client.0.smithi005.stderr:Locker ID Address 2024-09-13T07:30:23.742 INFO:tasks.workunit.client.0.smithi005.stderr:client.8528 id 172.21.15.5:0/3975735010' ']' 2024-09-13T07:30:23.743 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd lock list test1 2024-09-13T07:30:23.743 INFO:tasks.workunit.client.0.smithi005.stderr:+ tail -n 1 2024-09-13T07:30:23.743 INFO:tasks.workunit.client.0.smithi005.stderr:+ awk '{print $2, $1;}' 2024-09-13T07:30:23.743 INFO:tasks.workunit.client.0.smithi005.stderr:+ xargs rbd lock remove test1 2024-09-13T07:30:24.686 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd lock list test1 2024-09-13T07:30:24.751 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' -n '' ']' 2024-09-13T07:30:24.751 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:30:24.933 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:24.928+0000 7ff8bcaa9700 0 -- 172.21.15.5:0/3442056135 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff89c000fb0 msgr2=0x7ff89c00d8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:24.963 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:24.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_clone 2024-09-13T07:30:24.969 INFO:tasks.workunit.client.0.smithi005.stdout:testing clone... 2024-09-13T07:30:24.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing clone...' 2024-09-13T07:30:24.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:30:24.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:25.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:25.261 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:25.412 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:25.547 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:25.696 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:25.848 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:26.198 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:26.352 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:26.506 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:26.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:26.801 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:26.952 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:27.104 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:27.253 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:27.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:27.551 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:27.696 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:27.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create test1 --image-format 2 -s 1 2024-09-13T07:30:27.946 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@s1 2024-09-13T07:30:28.706 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:28.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect test1@s1 2024-09-13T07:30:28.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:30:29.800 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:30:29.809 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:30:32.719 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test1@s1 rbd2/clone 2024-09-13T07:30:32.836 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p rbd2 ls 2024-09-13T07:30:32.837 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep clone 2024-09-13T07:30:32.889 INFO:tasks.workunit.client.0.smithi005.stdout:clone 2024-09-13T07:30:32.889 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p rbd2 ls -l 2024-09-13T07:30:32.890 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep clone 2024-09-13T07:30:32.890 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1@s1 2024-09-13T07:30:32.972 INFO:tasks.workunit.client.0.smithi005.stdout:clone 1 MiB rbd/test1@s1 2 2024-09-13T07:30:32.972 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:32.973 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v clone 2024-09-13T07:30:33.023 INFO:tasks.workunit.client.0.smithi005.stdout:test1 2024-09-13T07:30:33.023 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd flatten rbd2/clone 2024-09-13T07:30:33.133 INFO:tasks.workunit.client.0.smithi005.stderr: Image flatten: 100% complete...done. 2024-09-13T07:30:33.151 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create rbd2/clone@s1 2024-09-13T07:30:33.734 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:33.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect rbd2/clone@s1 2024-09-13T07:30:33.856 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/clone@s1 clone2 2024-09-13T07:30:33.990 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:33.990 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep clone2 2024-09-13T07:30:34.040 INFO:tasks.workunit.client.0.smithi005.stdout:clone2 2024-09-13T07:30:34.041 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:30:34.041 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep clone2 2024-09-13T07:30:34.041 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/clone@s1 2024-09-13T07:30:34.124 INFO:tasks.workunit.client.0.smithi005.stdout:clone2 1 MiB rbd2/clone@s1 2 2024-09-13T07:30:34.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd -p rbd2 ls 2024-09-13T07:30:34.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v clone2 2024-09-13T07:30:34.174 INFO:tasks.workunit.client.0.smithi005.stdout:clone 2024-09-13T07:30:34.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm clone2 2024-09-13T07:30:34.368 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:34.364+0000 7f5658323700 0 -- 172.21.15.5:0/226717826 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564cce387100 msgr2=0x564cce39f310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:34.397 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:34.402 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect rbd2/clone@s1 2024-09-13T07:30:34.516 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm rbd2/clone@s1 2024-09-13T07:30:34.733 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:30:34.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd2/clone 2024-09-13T07:30:34.910 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:34.908+0000 7f382a15d700 0 -- 172.21.15.5:0/1291242499 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x55e8941943f0 msgr2=0x55e894196880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:34.928 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:34.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect test1@s1 2024-09-13T07:30:35.059 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm test1@s1 2024-09-13T07:30:35.740 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:30:35.758 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:30:35.930 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:35.928+0000 7f6e8e036700 0 -- 172.21.15.5:0/2060967117 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5574d44b6240 msgr2=0x5574d44cc440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:35.965 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:35.971 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:30:36.844 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:30:36.857 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_trash 2024-09-13T07:30:36.857 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing trash...' 2024-09-13T07:30:36.858 INFO:tasks.workunit.client.0.smithi005.stdout:testing trash... 2024-09-13T07:30:36.858 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:30:36.858 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.000 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.144 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.295 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.446 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.600 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.742 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:37.888 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.039 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.192 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.332 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.786 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:38.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:39.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:39.223 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:39.372 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:39.521 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:30:39.614 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test2 2024-09-13T07:30:39.703 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:39.703 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:30:39.750 INFO:tasks.workunit.client.0.smithi005.stdout:test1 2024-09-13T07:30:39.750 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:39.751 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:30:39.799 INFO:tasks.workunit.client.0.smithi005.stdout:test2 2024-09-13T07:30:39.800 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:39.800 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:39.800 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:30:39.848 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:30:39.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:30:39.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*2.*' 2024-09-13T07:30:39.915 INFO:tasks.workunit.client.0.smithi005.stdout:test1 1 MiB 2 2024-09-13T07:30:39.916 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:30:39.916 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*2.*' 2024-09-13T07:30:39.980 INFO:tasks.workunit.client.0.smithi005.stdout:test2 1 MiB 2 2024-09-13T07:30:39.981 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv test1 2024-09-13T07:30:40.090 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:40.090 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:30:40.135 INFO:tasks.workunit.client.0.smithi005.stdout:test2 2024-09-13T07:30:40.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:40.136 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:40.136 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:40.185 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:30:40.186 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*2.*' 2024-09-13T07:30:40.248 INFO:tasks.workunit.client.0.smithi005.stdout:test2 1 MiB 2 2024-09-13T07:30:40.249 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:40.249 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:30:40.296 INFO:tasks.workunit.client.0.smithi005.stdout:22a2a0fc5208 test1 2024-09-13T07:30:40.296 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:40.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:40.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:40.344 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:40.345 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -l 2024-09-13T07:30:40.345 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*USER.*' 2024-09-13T07:30:40.406 INFO:tasks.workunit.client.0.smithi005.stdout:22a2a0fc5208 test1 USER Fri Sep 13 07:30:40 2024 expired at Fri Sep 13 07:30:40 2024 2024-09-13T07:30:40.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -l 2024-09-13T07:30:40.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v 'protected until' 2024-09-13T07:30:40.474 INFO:tasks.workunit.client.0.smithi005.stdout:ID NAME SOURCE DELETED_AT STATUS PARENT 2024-09-13T07:30:40.474 INFO:tasks.workunit.client.0.smithi005.stdout:22a2a0fc5208 test1 USER Fri Sep 13 07:30:40 2024 expired at Fri Sep 13 07:30:40 2024 2024-09-13T07:30:40.475 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:30:40.475 INFO:tasks.workunit.client.0.smithi005.stderr:++ cut -d ' ' -f 1 2024-09-13T07:30:40.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=22a2a0fc5208 2024-09-13T07:30:40.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm 22a2a0fc5208 2024-09-13T07:30:40.651 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:40.656 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv test2 2024-09-13T07:30:40.762 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:30:40.762 INFO:tasks.workunit.client.0.smithi005.stderr:++ cut -d ' ' -f 1 2024-09-13T07:30:40.815 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=22a5fc3c1cf1 2024-09-13T07:30:40.815 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info --image-id 22a5fc3c1cf1 2024-09-13T07:30:40.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd image '\''test2'\''' 2024-09-13T07:30:40.880 INFO:tasks.workunit.client.0.smithi005.stdout:rbd image 'test2': 2024-09-13T07:30:40.881 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd children --image-id 22a5fc3c1cf1 2024-09-13T07:30:40.881 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:40.881 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:40.947 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:40.947 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash restore 22a5fc3c1cf1 2024-09-13T07:30:41.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:41.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:30:41.063 INFO:tasks.workunit.client.0.smithi005.stdout:test2 2024-09-13T07:30:41.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:30:41.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:41.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:41.114 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:41.115 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls -l 2024-09-13T07:30:41.115 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*2.*' 2024-09-13T07:30:41.178 INFO:tasks.workunit.client.0.smithi005.stdout:test2 1 MiB 2 2024-09-13T07:30:41.178 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv test2 --expires-at '3600 sec' 2024-09-13T07:30:41.281 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: image test2 will expire at 2024-09-13T08:30:41.228751+0000 2024-09-13T07:30:41.287 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:41.288 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test2 2024-09-13T07:30:41.336 INFO:tasks.workunit.client.0.smithi005.stdout:22a5fc3c1cf1 test2 2024-09-13T07:30:41.336 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:41.337 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:41.337 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:41.388 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:41.389 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -l 2024-09-13T07:30:41.389 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test2.*USER.*protected until' 2024-09-13T07:30:41.452 INFO:tasks.workunit.client.0.smithi005.stdout:22a5fc3c1cf1 test2 USER Fri Sep 13 07:30:41 2024 protected until Fri Sep 13 08:30:41 2024 2024-09-13T07:30:41.452 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm 22a5fc3c1cf1 2024-09-13T07:30:41.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'Deferment time has not expired' 2024-09-13T07:30:41.504 INFO:tasks.workunit.client.0.smithi005.stdout:Deferment time has not expired, please use --force if you really want to remove the image 2024-09-13T07:30:41.505 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm --image-id 22a5fc3c1cf1 --force 2024-09-13T07:30:41.631 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:41.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:30:41.726 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@snap1 2024-09-13T07:30:42.685 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:42.705 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect test1@snap1 2024-09-13T07:30:42.829 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test1@snap1 clone 2024-09-13T07:30:43.171 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv test1 2024-09-13T07:30:43.286 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:43.286 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:30:43.337 INFO:tasks.workunit.client.0.smithi005.stdout:22ffc5faab5c test1 2024-09-13T07:30:43.337 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:43.338 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:43.338 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:43.385 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:43.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -l 2024-09-13T07:30:43.386 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*USER.*' 2024-09-13T07:30:43.446 INFO:tasks.workunit.client.0.smithi005.stdout:22ffc5faab5c test1 USER Fri Sep 13 07:30:43 2024 expired at Fri Sep 13 07:30:43 2024 2024-09-13T07:30:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -l 2024-09-13T07:30:43.447 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v 'protected until' 2024-09-13T07:30:43.510 INFO:tasks.workunit.client.0.smithi005.stdout:ID NAME SOURCE DELETED_AT STATUS PARENT 2024-09-13T07:30:43.510 INFO:tasks.workunit.client.0.smithi005.stdout:22ffc5faab5c test1 USER Fri Sep 13 07:30:43 2024 expired at Fri Sep 13 07:30:43 2024 2024-09-13T07:30:43.511 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:30:43.511 INFO:tasks.workunit.client.0.smithi005.stderr:++ cut -d ' ' -f 1 2024-09-13T07:30:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=22ffc5faab5c 2024-09-13T07:30:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls --image-id 22ffc5faab5c 2024-09-13T07:30:43.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:30:43.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:43.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:43.631 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:43.632 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls --image-id 22ffc5faab5c 2024-09-13T07:30:43.632 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '.*snap1.*' 2024-09-13T07:30:43.699 INFO:tasks.workunit.client.0.smithi005.stdout: 38 snap1 1 MiB yes Fri Sep 13 07:30:42 2024 2024-09-13T07:30:43.699 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd children --image-id 22ffc5faab5c 2024-09-13T07:30:43.699 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:43.700 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:43.786 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:43.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd children --image-id 22ffc5faab5c 2024-09-13T07:30:43.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep clone 2024-09-13T07:30:43.871 INFO:tasks.workunit.client.0.smithi005.stdout:rbd/clone 2024-09-13T07:30:43.872 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm clone 2024-09-13T07:30:44.051 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:44.048+0000 7f72dbfff700 0 -- 172.21.15.5:0/1506277518 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f72c0008be0 msgr2=0x7f72c000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:44.084 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:44.089 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect --image-id 22ffc5faab5c --snap snap1 2024-09-13T07:30:44.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm --image-id 22ffc5faab5c --snap snap1 2024-09-13T07:30:44.678 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:30:44.697 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls --image-id 22ffc5faab5c 2024-09-13T07:30:44.698 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:30:44.698 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:44.698 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:44.763 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:44.764 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash restore 22ffc5faab5c 2024-09-13T07:30:44.827 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@snap1 2024-09-13T07:30:45.679 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:45.700 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@snap2 2024-09-13T07:30:46.693 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls --image-id 22ffc5faab5c 2024-09-13T07:30:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:30:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:46.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:30:46.779 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:30:46.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge --image-id 22ffc5faab5c 2024-09-13T07:30:48.693 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:30:48.706 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:30:48.704+0000 7fe12503a700 0 -- 172.21.15.5:0/3010345293 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55618b54ff70 msgr2=0x55618b5b72e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:48.718 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls --image-id 22ffc5faab5c 2024-09-13T07:30:48.718 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:30:48.718 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:48.718 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:48.777 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:48.777 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_to_trash_on_remove=true --rbd_move_to_trash_on_remove_expire_seconds=3600 test1 2024-09-13T07:30:48.878 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:48.885 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:48.885 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:30:48.935 INFO:tasks.workunit.client.0.smithi005.stdout:22ffc5faab5c test1 2024-09-13T07:30:48.936 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:48.936 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:48.981 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:48.981 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -l 2024-09-13T07:30:48.981 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'test1.*USER.*protected until' 2024-09-13T07:30:49.041 INFO:tasks.workunit.client.0.smithi005.stdout:22ffc5faab5c test1 USER Fri Sep 13 07:30:48 2024 protected until Fri Sep 13 08:30:48 2024 2024-09-13T07:30:49.041 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm 22ffc5faab5c 2024-09-13T07:30:49.042 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'Deferment time has not expired' 2024-09-13T07:30:49.088 INFO:tasks.workunit.client.0.smithi005.stdout:Deferment time has not expired, please use --force if you really want to remove the image 2024-09-13T07:30:49.089 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm --image-id 22ffc5faab5c --force 2024-09-13T07:30:49.217 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:30:49.222 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:30:49.222 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:49.367 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:49.517 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:49.869 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.156 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.449 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.601 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.758 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:50.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:51.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:51.212 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:51.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:51.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:51.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.172 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.321 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_purge 2024-09-13T07:30:52.321 INFO:tasks.workunit.client.0.smithi005.stdout:testing trash purge... 2024-09-13T07:30:52.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing trash purge...' 2024-09-13T07:30:52.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:30:52.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.477 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.628 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:52.920 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.073 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.221 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.373 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.678 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.826 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:53.979 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:54.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:54.287 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:54.434 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:54.576 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:54.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:54.875 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:30:55.028 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:55.028 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:55.028 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:55.082 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:55.083 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:30:55.128 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete...done. 2024-09-13T07:30:55.134 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:30:55.220 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-09-13T07:30:55.308 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 2024-09-13T07:30:55.411 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 2024-09-13T07:30:55.520 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:55.520 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:55.521 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:30:55.570 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:30:55.570 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:30:55.751 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 50% complete...2024-09-13T07:30:55.747+0000 7fa1f8809700 0 -- 172.21.15.5:0/4173948371 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fa1dc004780 msgr2=0x7fa1dc008740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:30:55.769 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:30:55.773 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:55.773 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:55.773 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:55.823 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:55.823 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:30:55.913 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-09-13T07:30:56.003 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 --expires-at '1 hour' 2024-09-13T07:30:56.102 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: image testimg1 will expire at 2024-09-13T08:30:56.046936+0000 2024-09-13T07:30:56.107 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 --expires-at '3 hours' 2024-09-13T07:30:56.211 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: image testimg2 will expire at 2024-09-13T10:30:56.157782+0000 2024-09-13T07:30:56.219 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:56.219 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:56.219 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:30:56.268 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:30:56.269 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:30:56.314 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete...done. 2024-09-13T07:30:56.318 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:56.319 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:56.319 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:30:56.368 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:30:56.368 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge --expired-before 'now + 2 hours' 2024-09-13T07:30:56.503 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:30:56.510 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:56.510 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:56.510 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:56.562 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:56.562 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:56.563 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:30:56.610 INFO:tasks.workunit.client.0.smithi005.stdout:244ed19e655a testimg2 2024-09-13T07:30:56.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge --expired-before 'now + 4 hours' 2024-09-13T07:30:56.747 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:30:56.754 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:56.755 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:56.755 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:56.803 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:56.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:30:56.891 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1@snap 2024-09-13T07:30:57.713 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:30:57.732 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-09-13T07:30:57.824 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:30:57.909 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 2024-09-13T07:30:58.008 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 2024-09-13T07:30:58.113 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:30:58.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:58.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:58.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:30:58.275 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:30:58.276 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:30:58.276 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:30:58.573 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:30:58.574 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:58.574 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:58.574 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:30:58.622 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:30:58.623 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:58.623 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:30:58.671 INFO:tasks.workunit.client.0.smithi005.stdout:246f22515aab testimg1 2024-09-13T07:30:58.672 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:30:58.672 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{ print $1 }' 2024-09-13T07:30:58.720 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=246f22515aab 2024-09-13T07:30:58.721 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge --image-id 246f22515aab 2024-09-13T07:30:59.718 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:30:59.733 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:30:59.861 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:30:59.868 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:30:59.868 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:30:59.868 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:30:59.914 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:30:59.915 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:31:00.002 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-09-13T07:31:00.097 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap 2024-09-13T07:31:00.737 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:00.756 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:31:00.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 2024-09-13T07:31:00.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 2024-09-13T07:31:01.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:31:01.179 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:01.179 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:01.179 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:01.227 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:01.228 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:01.228 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:01.535 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:01.535 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:01.536 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:01.536 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:31:01.584 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:31:01.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:01.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:01.632 INFO:tasks.workunit.client.0.smithi005.stdout:249f5d7f7d8d testimg2 2024-09-13T07:31:01.633 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:31:01.633 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{ print $1 }' 2024-09-13T07:31:01.683 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=249f5d7f7d8d 2024-09-13T07:31:01.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge --image-id 249f5d7f7d8d 2024-09-13T07:31:02.730 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:31:02.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:02.875 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:31:02.882 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:02.882 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:02.882 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:31:02.931 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:31:02.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:31:03.026 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg2 2024-09-13T07:31:03.115 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:31:03.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg3@snap 2024-09-13T07:31:03.740 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:03.760 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 2024-09-13T07:31:03.870 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 2024-09-13T07:31:03.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:31:04.080 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:04.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:04.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:04.132 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:04.132 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:04.133 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:04.434 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:04.434 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:04.434 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:04.435 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:31:04.481 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:31:04.482 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:04.482 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg3 2024-09-13T07:31:04.530 INFO:tasks.workunit.client.0.smithi005.stdout:24cfe5d1d9ab testimg3 2024-09-13T07:31:04.531 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:31:04.531 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{ print $1 }' 2024-09-13T07:31:04.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=24cfe5d1d9ab 2024-09-13T07:31:04.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge --image-id 24cfe5d1d9ab 2024-09-13T07:31:04.743 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:31:04.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:04.891 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:31:04.896 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:04.897 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:04.897 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:31:04.945 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:31:04.945 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:31:05.028 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1@snap 2024-09-13T07:31:05.760 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:05.780 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-09-13T07:31:05.915 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg1@snap 2024-09-13T07:31:06.007 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:06.026 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:31:06.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap 2024-09-13T07:31:06.768 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:06.789 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-09-13T07:31:06.932 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-09-13T07:31:07.089 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg2@snap 2024-09-13T07:31:07.212 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:07.232 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg4@snap 2024-09-13T07:31:07.762 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:07.771 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:07.767+0000 7f4df8b7e700 0 -- 172.21.15.5:0/4097331975 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ca04dc1b90 msgr2=0x55ca04dc3f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:07.785 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-09-13T07:31:07.947 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg4@snap 2024-09-13T07:31:08.075 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:08.093 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 2024-09-13T07:31:08.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 2024-09-13T07:31:08.302 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:31:08.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg4 2024-09-13T07:31:08.515 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:08.515 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:08.515 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:08.564 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:08.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:08.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:09.187 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:09.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:09.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:09.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:09.237 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:09.238 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:09.238 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:31:09.282 INFO:tasks.workunit.client.0.smithi005.stdout:24f6d6f93227 testimg1 2024-09-13T07:31:09.283 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:09.283 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:09.330 INFO:tasks.workunit.client.0.smithi005.stdout:24fc49af651b testimg2 2024-09-13T07:31:09.331 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:09.331 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg4 2024-09-13T07:31:09.379 INFO:tasks.workunit.client.0.smithi005.stdout:2508ceed67a7 testimg4 2024-09-13T07:31:09.379 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg6 2024-09-13T07:31:09.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:09.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:09.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:09.527 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:09.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:09.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:10.198 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:10.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:10.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:10.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:31:10.245 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:31:10.246 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:10.246 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:31:10.288 INFO:tasks.workunit.client.0.smithi005.stdout:24f6d6f93227 testimg1 2024-09-13T07:31:10.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:10.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:10.338 INFO:tasks.workunit.client.0.smithi005.stdout:24fc49af651b testimg2 2024-09-13T07:31:10.338 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg5 2024-09-13T07:31:10.449 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:10.449 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:10.449 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:10.494 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:10.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:10.654 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:10.651+0000 7f89c5015700 0 -- 172.21.15.5:0/2846068516 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562b940de0d0 msgr2=0x562b940f62c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:10.679 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:10.675+0000 7f89c3db3700 0 -- 172.21.15.5:0/2846068516 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f89a0004f90 msgr2=0x7f89a0008740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:12.810 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 33% complete... Removing images: 66% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:31:12.817 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:12.817 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:12.817 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:31:12.868 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:31:12.869 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:31:12.957 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1@snap 2024-09-13T07:31:13.694 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:13.713 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-09-13T07:31:13.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg1@snap 2024-09-13T07:31:13.958 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:13.979 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:31:14.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg3@snap 2024-09-13T07:31:14.702 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:14.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap 2024-09-13T07:31:15.705 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:15.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-09-13T07:31:15.884 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-09-13T07:31:16.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg2@snap 2024-09-13T07:31:16.155 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:16.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg4@snap 2024-09-13T07:31:16.717 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete...2024-09-13T07:31:16.711+0000 7f477475b700 0 -- 172.21.15.5:0/2845735226 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f4750008be0 msgr2=0x7f475000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:16.718 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 100% complete...done. 2024-09-13T07:31:16.737 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-09-13T07:31:16.899 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg4@snap 2024-09-13T07:31:17.032 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:17.052 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg1 2024-09-13T07:31:17.162 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg2 2024-09-13T07:31:17.269 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:31:17.381 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg4 2024-09-13T07:31:17.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:17.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:17.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:17.539 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:17.540 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:17.540 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:17.770 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:17.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:17.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:17.771 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:17.819 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:17.819 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg6 2024-09-13T07:31:17.925 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:17.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:17.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 5 2024-09-13T07:31:17.974 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-09-13T07:31:17.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:17.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:19.116 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:19.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:19.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:19.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:19.165 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:19.166 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:19.166 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:31:19.212 INFO:tasks.workunit.client.0.smithi005.stdout:25567a30478f testimg1 2024-09-13T07:31:19.213 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:19.213 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:19.261 INFO:tasks.workunit.client.0.smithi005.stdout:255c81eee42d testimg2 2024-09-13T07:31:19.262 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:19.262 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg3 2024-09-13T07:31:19.310 INFO:tasks.workunit.client.0.smithi005.stdout:25626cad98c5 testimg3 2024-09-13T07:31:19.311 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg5 2024-09-13T07:31:19.416 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:19.416 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:19.416 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:19.466 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:19.467 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:19.467 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:21.979 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:31:22.030 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:31:22.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:22.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg3 2024-09-13T07:31:22.079 INFO:tasks.workunit.client.0.smithi005.stdout:25626cad98c5 testimg3 2024-09-13T07:31:22.080 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:31:22.080 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{ print $1 }' 2024-09-13T07:31:22.131 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=25626cad98c5 2024-09-13T07:31:22.131 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge --image-id 25626cad98c5 2024-09-13T07:31:22.719 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:31:22.740 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:22.871 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:31:22.877 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:22.877 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:31:22.877 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:22.926 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:31:22.927 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:31:23.020 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1@snap 2024-09-13T07:31:23.735 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:23.754 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-09-13T07:31:23.899 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg1@snap 2024-09-13T07:31:24.004 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:24.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:31:24.115 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap 2024-09-13T07:31:24.739 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:24.758 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-09-13T07:31:24.908 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-09-13T07:31:25.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg2@snap 2024-09-13T07:31:25.302 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:25.299+0000 7f6505a31700 0 --2- 172.21.15.5:0/2670218432 >> v2:172.21.15.5:3300/0 conn(0x5598aa4b5ae0 0x5598aa49e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:31:25.389 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:25.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg4@snap 2024-09-13T07:31:25.745 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:25.752 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:25.747+0000 7f5ab628b700 0 -- 172.21.15.5:0/2811761476 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f5a9c008be0 msgr2=0x7f5a9c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:25.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-09-13T07:31:25.930 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg4@snap 2024-09-13T07:31:26.065 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:26.085 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg1 2024-09-13T07:31:26.223 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:26.228 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg2 2024-09-13T07:31:26.371 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:26.377 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:31:26.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg4 2024-09-13T07:31:26.634 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:26.641 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:26.689 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:26.690 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:26.690 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:27.102 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:27.103 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:27.155 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:27.156 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:27.156 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:31:27.206 INFO:tasks.workunit.client.0.smithi005.stdout:25c24f66f615 testimg1 2024-09-13T07:31:27.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:27.207 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:27.261 INFO:tasks.workunit.client.0.smithi005.stdout:25c817fe6def testimg2 2024-09-13T07:31:27.261 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:27.261 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg4 2024-09-13T07:31:27.311 INFO:tasks.workunit.client.0.smithi005.stdout:25d485e1d3ff testimg4 2024-09-13T07:31:27.311 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg6 2024-09-13T07:31:27.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:27.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:27.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:27.470 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:27.471 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:27.471 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:28.013 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:28.013 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:28.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:28.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2 2024-09-13T07:31:28.064 INFO:tasks.workunit.client.0.smithi005.stdout:2 2024-09-13T07:31:28.065 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:28.065 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:31:28.113 INFO:tasks.workunit.client.0.smithi005.stdout:25c24f66f615 testimg1 2024-09-13T07:31:28.113 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:28.113 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:28.164 INFO:tasks.workunit.client.0.smithi005.stdout:25c817fe6def testimg2 2024-09-13T07:31:28.164 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg5 2024-09-13T07:31:28.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:28.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:28.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:28.328 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:28.329 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:28.502 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:28.499+0000 7f7824442700 0 -- 172.21.15.5:0/2198397894 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55fe41959f20 msgr2=0x55fe41972080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:28.520 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:28.515+0000 7f78229df700 0 -- 172.21.15.5:0/2198397894 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f780005f6c0 msgr2=0x7f7800061ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:29.974 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 33% complete... Removing images: 66% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:31:29.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:29.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:29.981 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:31:30.029 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:31:30.030 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg1 2024-09-13T07:31:30.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1@snap 2024-09-13T07:31:30.689 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:30.710 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2024-09-13T07:31:30.846 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg1@snap 2024-09-13T07:31:30.955 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:30.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 256 testimg3 2024-09-13T07:31:31.066 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg3@snap 2024-09-13T07:31:31.848 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:31.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap 2024-09-13T07:31:32.845 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:32.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2024-09-13T07:31:33.008 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2024-09-13T07:31:33.158 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg2@snap 2024-09-13T07:31:33.276 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:33.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg4@snap 2024-09-13T07:31:33.854 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:33.861 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:33.855+0000 7fe564106700 0 -- 172.21.15.5:0/1085491788 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x555c69cb7680 msgr2=0x555c69d1f940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:33.876 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2024-09-13T07:31:34.044 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm testimg4@snap 2024-09-13T07:31:34.176 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:31:34.198 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg1 2024-09-13T07:31:34.343 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:34.348 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg2 2024-09-13T07:31:34.495 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:34.501 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg3 2024-09-13T07:31:34.818 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg4 2024-09-13T07:31:34.968 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:34.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:34.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:34.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:35.026 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:35.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:35.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:35.273 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:35.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:35.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:35.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:35.321 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:35.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg6 2024-09-13T07:31:35.431 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:35.431 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 5 2024-09-13T07:31:35.431 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:35.480 INFO:tasks.workunit.client.0.smithi005.stdout:5 2024-09-13T07:31:35.480 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:35.480 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:36.099 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:36.100 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:36.100 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:36.100 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 3 2024-09-13T07:31:36.148 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:31:36.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:36.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg1 2024-09-13T07:31:36.198 INFO:tasks.workunit.client.0.smithi005.stdout:262189d8894a testimg1 2024-09-13T07:31:36.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:36.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg2 2024-09-13T07:31:36.247 INFO:tasks.workunit.client.0.smithi005.stdout:2627293fb5ca testimg2 2024-09-13T07:31:36.247 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:36.247 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg3 2024-09-13T07:31:36.294 INFO:tasks.workunit.client.0.smithi005.stdout:262dcbf7a5e3 testimg3 2024-09-13T07:31:36.295 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv testimg5 2024-09-13T07:31:36.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:36.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 4 2024-09-13T07:31:36.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:36.458 INFO:tasks.workunit.client.0.smithi005.stdout:4 2024-09-13T07:31:36.458 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:36.458 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'some expired images could not be removed' 2024-09-13T07:31:39.069 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: some expired images could not be removed 2024-09-13T07:31:39.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:39.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:39.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:31:39.119 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:31:39.119 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:39.120 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep testimg3 2024-09-13T07:31:39.166 INFO:tasks.workunit.client.0.smithi005.stdout:262dcbf7a5e3 testimg3 2024-09-13T07:31:39.167 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash ls 2024-09-13T07:31:39.167 INFO:tasks.workunit.client.0.smithi005.stderr:++ awk '{ print $1 }' 2024-09-13T07:31:39.213 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=262dcbf7a5e3 2024-09-13T07:31:39.213 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge --image-id 262dcbf7a5e3 2024-09-13T07:31:39.865 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:31:39.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge 2024-09-13T07:31:40.012 INFO:tasks.workunit.client.0.smithi005.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2024-09-13T07:31:40.020 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls 2024-09-13T07:31:40.020 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:40.020 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 0 2024-09-13T07:31:40.062 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:31:40.062 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_deep_copy_clone 2024-09-13T07:31:40.063 INFO:tasks.workunit.client.0.smithi005.stdout:testing deep copy clone... 2024-09-13T07:31:40.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing deep copy clone...' 2024-09-13T07:31:40.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:31:40.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:40.210 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:40.358 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:40.513 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:40.663 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:40.818 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:40.971 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:41.119 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:41.267 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:41.417 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:41.575 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:41.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:41.882 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:42.037 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:42.190 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:42.340 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:42.492 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:42.641 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:42.786 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create testimg1 --image-format 2 --size 256 2024-09-13T07:31:42.881 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg1 --snap=snap1 2024-09-13T07:31:43.361 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:43.379 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect testimg1@snap1 2024-09-13T07:31:43.505 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone testimg1@snap1 testimg2 2024-09-13T07:31:43.650 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap2 2024-09-13T07:31:44.361 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:44.381 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep copy testimg2 testimg3 2024-09-13T07:31:44.553 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:44.550+0000 7f92414e6700 0 -- 172.21.15.5:0/3121605571 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x564ffa559cd0 msgr2=0x564ffa5bb840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:31:44.580 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:44.578+0000 7f92414e6700 0 -- 172.21.15.5:0/3121605571 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f9224039f70 msgr2=0x7f922403c3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:45.389 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-09-13T07:31:45.396 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:31:45.396 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:31:45.478 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:31:45.478 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:31:45.478 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: rbd/testimg1@snap1' 2024-09-13T07:31:45.563 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd/testimg1@snap1 2024-09-13T07:31:45.563 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg3 2024-09-13T07:31:45.563 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:31:45.564 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:45.564 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:31:45.642 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:31:45.643 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg3 2024-09-13T07:31:45.643 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '.*snap2.*' 2024-09-13T07:31:45.723 INFO:tasks.workunit.client.0.smithi005.stdout: 80 snap2 256 MiB Fri Sep 13 07:31:45 2024 2024-09-13T07:31:45.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg2 2024-09-13T07:31:45.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'features:.*deep-flatten' 2024-09-13T07:31:45.807 INFO:tasks.workunit.client.0.smithi005.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling, data-pool 2024-09-13T07:31:45.807 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:31:45.807 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'features:.*deep-flatten' 2024-09-13T07:31:45.890 INFO:tasks.workunit.client.0.smithi005.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling, data-pool 2024-09-13T07:31:45.890 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd flatten testimg2 2024-09-13T07:31:46.015 INFO:tasks.workunit.client.0.smithi005.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2024-09-13T07:31:46.036 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd flatten testimg3 2024-09-13T07:31:46.160 INFO:tasks.workunit.client.0.smithi005.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2024-09-13T07:31:46.180 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect testimg1@snap1 2024-09-13T07:31:46.312 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge testimg2 2024-09-13T07:31:47.371 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:31:47.392 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge testimg3 2024-09-13T07:31:48.376 INFO:tasks.workunit.client.0.smithi005.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T07:31:48.395 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm testimg2 2024-09-13T07:31:48.591 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-09-13T07:31:48.590+0000 7fe10238f700 0 -- 172.21.15.5:0/3694488827 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x557f2a419100 msgr2=0x557f2a431800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:48.625 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:48.631 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm testimg3 2024-09-13T07:31:48.823 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2024-09-13T07:31:48.822+0000 7f43fb35a700 0 -- 172.21.15.5:0/1075773235 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55ebb7bf1830 msgr2=0x55ebb7bf3c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:48.856 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:31:48.861 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect testimg1@snap1 2024-09-13T07:31:48.985 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone testimg1@snap1 testimg2 2024-09-13T07:31:49.123 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create testimg2@snap2 2024-09-13T07:31:49.394 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:31:49.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd deep copy --flatten testimg2 testimg3 2024-09-13T07:31:50.390 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:50.390+0000 7fb5a6f5c700 0 -- 172.21.15.5:0/1500930301 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5595ccc39140 msgr2=0x5595ccc9d6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:50.403 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:31:50.402+0000 7fb5a6f5c700 0 -- 172.21.15.5:0/1500930301 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fb58803a530 msgr2=0x7fb58803c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:31:50.806 INFO:tasks.workunit.client.0.smithi005.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2024-09-13T07:31:50.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:31:50.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'size 256 MiB' 2024-09-13T07:31:50.874 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:31:50.875 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg3 2024-09-13T07:31:50.875 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v parent: 2024-09-13T07:31:50.938 INFO:tasks.workunit.client.0.smithi005.stdout:rbd image 'testimg3': 2024-09-13T07:31:50.939 INFO:tasks.workunit.client.0.smithi005.stdout: size 256 MiB in 64 objects 2024-09-13T07:31:50.939 INFO:tasks.workunit.client.0.smithi005.stdout: order 22 (4 MiB objects) 2024-09-13T07:31:50.939 INFO:tasks.workunit.client.0.smithi005.stdout: snapshot_count: 1 2024-09-13T07:31:50.939 INFO:tasks.workunit.client.0.smithi005.stdout: id: 273b56e6cb34 2024-09-13T07:31:50.939 INFO:tasks.workunit.client.0.smithi005.stdout: data_pool: datapool 2024-09-13T07:31:50.939 INFO:tasks.workunit.client.0.smithi005.stdout: block_name_prefix: rbd_data.2.273b56e6cb34 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: format: 2 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling, data-pool 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: op_features: 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: flags: 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: create_timestamp: Fri Sep 13 07:31:49 2024 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: access_timestamp: Fri Sep 13 07:31:49 2024 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: modify_timestamp: Fri Sep 13 07:31:49 2024 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: journal: 273b56e6cb34 2024-09-13T07:31:50.940 INFO:tasks.workunit.client.0.smithi005.stdout: mirroring state: disabled 2024-09-13T07:31:50.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg3 2024-09-13T07:31:50.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -v SNAPID 2024-09-13T07:31:50.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:31:50.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 1 2024-09-13T07:31:51.004 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:31:51.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap ls testimg3 2024-09-13T07:31:51.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '.*snap2.*' 2024-09-13T07:31:51.070 INFO:tasks.workunit.client.0.smithi005.stdout: 84 snap2 256 MiB Fri Sep 13 07:31:50 2024 2024-09-13T07:31:51.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info testimg2 2024-09-13T07:31:51.071 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'features:.*deep-flatten' 2024-09-13T07:31:51.153 INFO:tasks.workunit.client.0.smithi005.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling, data-pool 2024-09-13T07:31:51.153 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd flatten testimg2 2024-09-13T07:31:51.280 INFO:tasks.workunit.client.0.smithi005.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2024-09-13T07:31:51.303 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect testimg1@snap1 2024-09-13T07:31:51.432 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:31:51.433 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:52.641 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:53.659 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:54.715 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:54.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.016 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.328 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.481 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.629 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.782 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:55.934 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.085 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.238 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.532 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.690 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.848 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:56.997 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_clone_v2 2024-09-13T07:31:56.997 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing clone v2...' 2024-09-13T07:31:56.998 INFO:tasks.workunit.client.0.smithi005.stdout:testing clone v2... 2024-09-13T07:31:56.998 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:31:56.998 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:57.151 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:57.304 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:57.463 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:57.619 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:57.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:57.936 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:58.086 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:58.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:58.404 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:58.561 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:58.715 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:58.875 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:59.030 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:59.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:59.332 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:59.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:59.639 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:31:59.792 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:31:59.883 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@1 2024-09-13T07:32:00.427 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:00.449 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=1 test1@1 test2 2024-09-13T07:32:00.518 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:00.518+0000 7f461daa8700 -1 librbd::image::CloneRequest: 0x55d3e700cef0 validate_parent: parent snapshot must be protected 2024-09-13T07:32:00.518 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: clone error: (22) Invalid argument 2024-09-13T07:32:00.525 INFO:tasks.workunit.client.0.smithi005.stderr:+ true 2024-09-13T07:32:00.525 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 test1@1 test2 2024-09-13T07:32:00.662 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=2 test1@1 test3 2024-09-13T07:32:01.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect test1@1 2024-09-13T07:32:01.143 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format=1 test1@1 test4 2024-09-13T07:32:01.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd children test1@1 2024-09-13T07:32:01.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ sort 2024-09-13T07:32:01.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ tr '\n' ' ' 2024-09-13T07:32:01.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -E 'test2.*test3.*test4' 2024-09-13T07:32:01.374 INFO:tasks.workunit.client.0.smithi005.stdout:rbd/test2 rbd/test3 rbd/test4 2024-09-13T07:32:01.374 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd children --descendants test1 2024-09-13T07:32:01.374 INFO:tasks.workunit.client.0.smithi005.stderr:+ sort 2024-09-13T07:32:01.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ tr '\n' ' ' 2024-09-13T07:32:01.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -E 'test2.*test3.*test4' 2024-09-13T07:32:01.483 INFO:tasks.workunit.client.0.smithi005.stdout:rbd/test2 rbd/test3 rbd/test4 2024-09-13T07:32:01.483 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd remove test4 2024-09-13T07:32:01.676 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:01.674+0000 7fc75556c700 0 -- 172.21.15.5:0/2369432616 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fc730057950 msgr2=0x7fc730059a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:01.914 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:01.919 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect test1@1 2024-09-13T07:32:02.049 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap remove test1@1 2024-09-13T07:32:02.152 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:02.172 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap list --all test1 2024-09-13T07:32:02.172 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep -E 'trash \(1\) *$' 2024-09-13T07:32:02.239 INFO:tasks.workunit.client.0.smithi005.stdout: 88 68fb1f91-b2ab-4011-a23e-572504401263 1 MiB Fri Sep 13 07:32:00 2024 trash (1) 2024-09-13T07:32:02.239 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@2 2024-09-13T07:32:02.432 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:02.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:02.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'image has snapshots' 2024-09-13T07:32:02.565 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed. 2024-09-13T07:32:02.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm test1@2 2024-09-13T07:32:03.427 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:03.446 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:03.446 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'linked clones' 2024-09-13T07:32:03.560 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-09-13T07:32:03.560 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test3 2024-09-13T07:32:03.760 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:03.758+0000 7f55a4187700 0 -- 172.21.15.5:0/3788138315 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f5584003820 msgr2=0x7f5584001bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:04.005 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:04.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:04.012 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'linked clones' 2024-09-13T07:32:04.123 INFO:tasks.workunit.client.0.smithi005.stdout:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-09-13T07:32:04.123 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd flatten test2 2024-09-13T07:32:04.433 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:04.430+0000 7f28c90e0700 0 -- 172.21.15.5:0/1794311716 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f28a4008be0 msgr2=0x7f28a400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:04.450 INFO:tasks.workunit.client.0.smithi005.stderr: Image flatten: 100% complete...done. 2024-09-13T07:32:04.460 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:04.458+0000 7f28c90e0700 0 -- 172.21.15.5:0/1794311716 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f28ac03a650 msgr2=0x7f28ac03ca40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:04.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap list --all test1 2024-09-13T07:32:04.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:32:04.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:32:04.546 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:32:04.546 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:04.732 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:04.730+0000 7f2c2afa7700 0 -- 172.21.15.5:0/3970134814 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f2c100163f0 msgr2=0x7f2c10012c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:04.768 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:04.774 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:32:04.966 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:04.966+0000 7f504ae71700 0 -- 172.21.15.5:0/3643647124 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55b342f2f380 msgr2=0x55b342f31770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:05.010 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:05.015 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 test1 2024-09-13T07:32:05.107 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@1 2024-09-13T07:32:05.442 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:05.463 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create test1@2 2024-09-13T07:32:06.456 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:06.476 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test1@1 test2 --rbd-default-clone-format 2 2024-09-13T07:32:06.615 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone test1@2 test3 --rbd-default-clone-format 2 2024-09-13T07:32:06.751 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm test1@1 2024-09-13T07:32:06.868 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:06.888 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm test1@2 2024-09-13T07:32:07.002 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:07.025 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd rm test1 2024-09-13T07:32:07.025 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:07.121 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:07.118+0000 7f7b75554780 -1 librbd::api::Image: remove: image has snapshots - not removing 2024-09-13T07:32:07.121 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 0% complete...failed. 2024-09-13T07:32:07.136 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-09-13T07:32:07.142 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:07.142 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 --rbd-move-parent-to-trash-on-remove=true 2024-09-13T07:32:07.295 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:07.300 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -a 2024-09-13T07:32:07.300 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:07.555 INFO:tasks.workunit.client.0.smithi005.stdout:286f5f07512d test1 2024-09-13T07:32:07.555 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test2 2024-09-13T07:32:07.794 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:07.794+0000 7ff51d0a4700 0 --2- 172.21.15.5:0/3920422983 >> v2:172.21.15.5:3300/0 conn(0x555724290100 0x55572427bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:32:07.931 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:07.930+0000 7ff51e306700 0 -- 172.21.15.5:0/2406992279 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7ff508001d50 msgr2=0x5557242a7bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:07.950 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:07.950+0000 7ff51c8a3700 0 -- 172.21.15.5:0/2406992279 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7ff4f8008be0 msgr2=0x7ff4f800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:08.503 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:08.509 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -a 2024-09-13T07:32:08.509 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:08.559 INFO:tasks.workunit.client.0.smithi005.stdout:286f5f07512d test1 2024-09-13T07:32:08.560 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test3 2024-09-13T07:32:08.740 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:08.738+0000 7f4028bd5700 0 -- 172.21.15.5:0/3776168060 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f4004012c90 msgr2=0x7f4004013110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:08.762 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:08.762+0000 7f402a638700 0 -- 172.21.15.5:0/3776168060 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55aa8343f0d0 msgr2=0x55aa834572c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:09.555 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:09.561 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls -a 2024-09-13T07:32:09.562 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep test1 2024-09-13T07:32:09.562 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:09.608 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:09.609 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_thick_provision 2024-09-13T07:32:09.609 INFO:tasks.workunit.client.0.smithi005.stdout:testing thick provision... 2024-09-13T07:32:09.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing thick provision...' 2024-09-13T07:32:09.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:32:09.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:09.759 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:09.914 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.219 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.369 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.528 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.680 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.837 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:10.988 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:11.145 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:11.299 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:11.456 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:11.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:11.769 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:11.922 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:12.070 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:12.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:12.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --thick-provision -s 64M test1 2024-09-13T07:32:12.612 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 6% complete... Thick provisioning: 12% complete... Thick provisioning: 18% complete... Thick provisioning: 25% complete... Thick provisioning: 31% complete... Thick provisioning: 37% complete... Thick provisioning: 43% complete... Thick provisioning: 50% complete... Thick provisioning: 56% complete... Thick provisioning: 62% complete... Thick provisioning: 68% complete... Thick provisioning: 75% complete... Thick provisioning: 81% complete... Thick provisioning: 87% complete... Thick provisioning: 93% complete... Thick provisioning: 100% complete...2024-09-13T07:32:12.610+0000 7f3df0403700 0 -- 172.21.15.5:0/2693789432 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f3dcc0055f0 msgr2=0x7f3dcc006f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:12.648 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 100% complete...done. 2024-09-13T07:32:12.664 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:12.662+0000 7f3df1e66700 0 -- 172.21.15.5:0/2693789432 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x561d02d87870 msgr2=0x561d02d89c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:12.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ count=0 2024-09-13T07:32:12.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret= 2024-09-13T07:32:12.669 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 -lt 10 ']' 2024-09-13T07:32:12.670 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:32:12.670 INFO:tasks.workunit.client.0.smithi005.stderr:+ tr -s ' ' 2024-09-13T07:32:12.670 INFO:tasks.workunit.client.0.smithi005.stderr:+ cut -d ' ' -f 4-5 2024-09-13T07:32:12.671 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:12.671 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^64 MiB' 2024-09-13T07:32:12.732 INFO:tasks.workunit.client.0.smithi005.stdout:64 MiB 2024-09-13T07:32:12.733 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret=0 2024-09-13T07:32:12.733 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 = 0 ']' 2024-09-13T07:32:12.733 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:32:12.733 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:32:12.790 INFO:tasks.workunit.client.0.smithi005.stdout:NAME PROVISIONED USED 2024-09-13T07:32:12.790 INFO:tasks.workunit.client.0.smithi005.stdout:test1 64 MiB 64 MiB 2024-09-13T07:32:12.795 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 '!=' 0 ']' 2024-09-13T07:32:12.795 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:12.977 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:12.974+0000 7f8d2484c700 0 -- 172.21.15.5:0/1449913538 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f8d04006670 msgr2=0x7f8d04008aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:13.009 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-13T07:32:13.006+0000 7f8d262af700 0 -- 172.21.15.5:0/1449913538 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f8d100056c0 msgr2=0x56277c9d9c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:13.051 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:13.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:32:13.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:13.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:32:13.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:32:13.109 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:32:13.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --thick-provision -s 4G test1 2024-09-13T07:32:13.305 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 1% complete...2024-09-13T07:32:13.302+0000 7f6a1fe7e700 0 -- 172.21.15.5:0/2525000985 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f6a0003a040 msgr2=0x7f6a0003c470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:13.347 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 2% complete...2024-09-13T07:32:13.346+0000 7f6a1fe7e700 0 -- 172.21.15.5:0/2525000985 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e830d068f0 msgr2=0x7f6a0003d860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:19.990 INFO:tasks.workunit.client.0.smithi005.stderr: Thick provisioning: 3% complete... Thick provisioning: 4% complete... Thick provisioning: 5% complete... Thick provisioning: 6% complete... Thick provisioning: 7% complete... Thick provisioning: 8% complete... Thick provisioning: 9% complete... Thick provisioning: 10% complete... Thick provisioning: 11% complete... Thick provisioning: 12% complete... Thick provisioning: 13% complete... Thick provisioning: 14% complete... Thick provisioning: 15% complete... Thick provisioning: 16% complete... Thick provisioning: 17% complete... Thick provisioning: 18% complete... Thick provisioning: 19% complete... Thick provisioning: 20% complete... Thick provisioning: 21% complete... Thick provisioning: 22% complete... Thick provisioning: 23% complete... Thick provisioning: 24% complete... Thick provisioning: 25% complete... Thick provisioning: 26% complete... Thick provisioning: 27% complete... Thick provisioning: 28% complete... Thick provisioning: 29% complete... Thick provisioning: 30% complete... Thick provisioning: 31% complete... Thick provisioning: 32% complete... Thick provisioning: 33% complete... Thick provisioning: 34% complete... Thick provisioning: 35% complete... Thick provisioning: 36% complete... Thick provisioning: 37% complete... Thick provisioning: 38% complete... Thick provisioning: 39% complete... Thick provisioning: 40% complete... Thick provisioning: 41% complete... Thick provisioning: 42% complete... Thick provisioning: 43% complete... Thick provisioning: 44% complete... Thick provisioning: 45% complete... Thick provisioning: 46% complete... Thick provisioning: 47% complete... Thick provisioning: 48% complete... Thick provisioning: 49% complete... Thick provisioning: 50% complete... Thick provisioning: 51% complete... Thick provisioning: 52% complete... Thick provisioning: 53% complete... Thick provisioning: 54% complete... Thick provisioning: 55% complete... Thick provisioning: 56% complete... Thick provisioning: 57% complete... Thick provisioning: 58% complete... Thick provisioning: 59% complete... Thick provisioning: 60% complete... Thick provisioning: 61% complete... Thick provisioning: 62% complete... Thick provisioning: 63% complete... Thick provisioning: 64% complete... Thick provisioning: 65% complete... Thick provisioning: 66% complete... Thick provisioning: 67% complete... Thick provisioning: 68% complete... Thick provisioning: 69% complete... Thick provisioning: 70% complete... Thick provisioning: 71% complete... Thick provisioning: 72% complete... Thick provisioning: 73% complete... Thick provisioning: 74% complete... Thick provisioning: 75% complete... Thick provisioning: 76% complete... Thick provisioning: 77% complete... Thick provisioning: 78% complete... Thick provisioning: 79% complete... Thick provisioning: 80% complete... Thick provisioning: 81% complete... Thick provisioning: 82% complete... Thick provisioning: 83% complete... Thick provisioning: 84% complete... Thick provisioning: 85% complete... Thick provisioning: 86% complete... Thick provisioning: 87% complete... Thick provisioning: 88% complete... Thick provisioning: 89% complete... Thick provisioning: 90% complete... Thick provisioning: 91% complete... Thick provisioning: 92% complete... Thick provisioning: 93% complete... Thick provisioning: 94% complete... Thick provisioning: 95% complete... Thick provisioning: 96% complete... Thick provisioning: 97% complete... Thick provisioning: 98% complete... Thick provisioning: 99% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2024-09-13T07:32:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ count=0 2024-09-13T07:32:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret= 2024-09-13T07:32:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 -lt 10 ']' 2024-09-13T07:32:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:32:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:20.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ tr -s ' ' 2024-09-13T07:32:20.010 INFO:tasks.workunit.client.0.smithi005.stderr:+ cut -d ' ' -f 4-5 2024-09-13T07:32:20.010 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^4 GiB' 2024-09-13T07:32:20.071 INFO:tasks.workunit.client.0.smithi005.stdout:4 GiB 2024-09-13T07:32:20.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ ret=0 2024-09-13T07:32:20.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 = 0 ']' 2024-09-13T07:32:20.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:32:20.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd du 2024-09-13T07:32:20.130 INFO:tasks.workunit.client.0.smithi005.stdout:NAME PROVISIONED USED 2024-09-13T07:32:20.130 INFO:tasks.workunit.client.0.smithi005.stdout:test1 4 GiB 4 GiB 2024-09-13T07:32:20.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ '[' 0 '!=' 0 ']' 2024-09-13T07:32:20.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm test1 2024-09-13T07:32:20.334 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete...2024-09-13T07:32:20.334+0000 7f1813cc4700 0 -- 172.21.15.5:0/4230286900 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x560d88cae220 msgr2=0x560d88cb06b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:20.359 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 2% complete... Removing image: 3% complete...2024-09-13T07:32:20.358+0000 7f1813cc4700 0 -- 172.21.15.5:0/4230286900 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f17f403a030 msgr2=0x7f17f403c460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:21.326 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T07:32:21.332 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd ls 2024-09-13T07:32:21.332 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test1 2024-09-13T07:32:21.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:32:21.333 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:32:21.379 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:32:21.380 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_namespace 2024-09-13T07:32:21.380 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing namespace...' 2024-09-13T07:32:21.380 INFO:tasks.workunit.client.0.smithi005.stdout:testing namespace... 2024-09-13T07:32:21.380 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:32:21.381 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:21.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:21.685 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:21.836 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:21.994 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:22.153 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:22.316 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:22.472 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:22.629 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:22.788 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:22.951 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:23.111 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:23.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:23.427 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:23.590 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:23.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:23.904 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:24.068 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:24.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace ls 2024-09-13T07:32:24.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:32:24.227 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:32:24.273 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:32:24.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd/test1 2024-09-13T07:32:24.339 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create --pool rbd --namespace test2 2024-09-13T07:32:24.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create --namespace test3 2024-09-13T07:32:24.474 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd namespace create rbd/test3 2024-09-13T07:32:24.475 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd/test3 2024-09-13T07:32:24.522 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: failed to created namespace: 2024-09-13T07:32:24.521+0000 7fc848b85780 -1 librbd::api::Namespace: create: failed to add namespace: (17) File exists 2024-09-13T07:32:24.523 INFO:tasks.workunit.client.0.smithi005.stderr:(17) File exists 2024-09-13T07:32:24.529 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:24.529 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace list 2024-09-13T07:32:24.529 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test 2024-09-13T07:32:24.530 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:32:24.530 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^3$' 2024-09-13T07:32:24.576 INFO:tasks.workunit.client.0.smithi005.stdout:3 2024-09-13T07:32:24.576 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd namespace remove --pool rbd missing 2024-09-13T07:32:24.576 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace remove --pool rbd missing 2024-09-13T07:32:24.608 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: namespace name was not specified 2024-09-13T07:32:24.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:24.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G rbd/test1/image1 2024-09-13T07:32:24.704 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 32M --io-size 4K rbd/test1/image1 2024-09-13T07:32:24.810 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 4096 io_threads 16 bytes 33554432 pattern random 2024-09-13T07:32:25.568 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:25.565+0000 7f41f0dbe700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f41c8004b40 msgr2=0x7f41d003f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:25.787 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:25.785+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55c7587510a0 msgr2=0x7f41d00496b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:26.280 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:32:26.280 INFO:tasks.workunit.client.0.smithi005.stdout: 1 3872 2655.8 10 MiB/s 2024-09-13T07:32:26.848 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:26.845+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f41d0039fc0 msgr2=0x7f41d00c6f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:26.975 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:26.973+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f41c8004b40 msgr2=0x7f41d003f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:27.318 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:27.317+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f41d0039fc0 msgr2=0x7f41d003c3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:28.057 INFO:tasks.workunit.client.0.smithi005.stdout: 3 6960 2153.14 8.4 MiB/s 2024-09-13T07:32:28.319 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:28.317+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f41d0039fc0 msgr2=0x7f41d003c3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:28.853 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:28.849+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f41c8004b40 msgr2=0x7f41d003f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:28.933 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:28.929+0000 7f41efb5c700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f41d0039fc0 msgr2=0x7f41d003c3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:29.148 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:29.145+0000 7f41f0dbe700 0 -- 172.21.15.5:0/2707581091 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f41d0039fc0 msgr2=0x7f41d003c3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:29.546 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 4 ops: 8192 ops/sec: 1729.77 bytes/sec: 6.8 MiB/s 2024-09-13T07:32:29.567 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create rbd/test1/image1@1 2024-09-13T07:32:30.709 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:30.726 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:30.725+0000 7f8d1fdd2700 0 -- 172.21.15.5:0/1344278841 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x559a65f5dae0 msgr2=0x559a65f73cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:30.734 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format 2 rbd/test1/image1@1 rbd/test2/image1 2024-09-13T07:32:30.876 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm rbd/test1/image1@1 2024-09-13T07:32:31.016 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:31.038 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export rbd/test1/image1 - 2024-09-13T07:32:31.038 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /dev/fd/63 /dev/fd/62 2024-09-13T07:32:31.039 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export rbd/test2/image1 - 2024-09-13T07:32:31.278 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: Exporting image4: % complete...4 % complete...Exporting image: 5% complete... Exporting image: 6Exporting image% complete...: 5% complete... Exporting image: 7% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting imageExporting image: : 118% complete...% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete...2024-09-13T07:32:31.277+0000 7fb30aadb700 0 -- 172.21.15.5:0/3620607625 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5573eda05ef0 msgr2=0x5573eda082e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:31.332 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete...2024-09-13T07:32:31.329+0000 7fb30a2da700 0 -- 172.21.15.5:0/3620607625 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fb2ec004be0 msgr2=0x7fb2ec0075b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:31.514 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: Exporting image25: % complete...24% complete... Exporting image : 26Exporting image% complete...: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete...2024-09-13T07:32:31.513+0000 7f78f208a700 0 -- 172.21.15.5:0/784946433 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x558cc5484df0 msgr2=0x558cc549a7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:31.626 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete...2024-09-13T07:32:31.625+0000 7f78f0e28700 0 -- 172.21.15.5:0/784946433 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f78d0004650 msgr2=0x7f78d0004ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:32.593 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting imageExporting image: : 4343% complete...% complete... Exporting image: 44 % complete...Exporting image: 44% complete... Exporting image: 45% complete... Exporting image : Exporting image45: % complete...46% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting imageExporting image: : 5555% complete...% complete... Exporting image: 56Exporting image% complete...: 56% complete... Exporting image: 57% complete... Exporting image: 57Exporting image% complete...: 58% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: Exporting image60: % complete...59% complete... Exporting image : Exporting image60: % complete...61% complete... Exporting image: 61Exporting image% complete...: 62% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting imageExporting image: : 7878% complete...% complete... Exporting image: 79 % complete...Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting imageExporting image: 100% complete...done.: 100% complete...done. 2024-09-13T07:32:32.593 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:32:32.599 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd/test2/image1 2024-09-13T07:32:32.983 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete...2024-09-13T07:32:32.981+0000 7fb162feb700 0 -- 172.21.15.5:0/684854241 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fb14400b9d0 msgr2=0x7fb144005c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:33.014 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:33.013+0000 7fb1637ec700 0 -- 172.21.15.5:0/684854241 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55cf82943750 msgr2=0x55cf82945be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:33.774 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:33.780 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G rbd/image2 2024-09-13T07:32:33.870 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 32M --io-size 4K rbd/image2 2024-09-13T07:32:33.970 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 4096 io_threads 16 bytes 33554432 pattern random 2024-09-13T07:32:34.046 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:34.045+0000 7f019bcbf700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f017c00b6e0 msgr2=0x7f017c00db50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:34.721 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:34.717+0000 7f019d722700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562536280400 msgr2=0x5625362e2e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:35.197 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:32:35.197 INFO:tasks.workunit.client.0.smithi005.stdout: 1 2928 2413.17 9.4 MiB/s 2024-09-13T07:32:35.484 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:35.481+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f017c004b90 msgr2=0x7f017c006f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:35.881 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:35.877+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f017c00b6e0 msgr2=0x7f0180050f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:36.077 INFO:tasks.workunit.client.0.smithi005.stdout: 2 4576 2186.72 8.5 MiB/s 2024-09-13T07:32:36.130 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:36.129+0000 7f019d722700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x562536280400 msgr2=0x7f0180058d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:36.299 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:36.297+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x562536280400 msgr2=0x7f0180058d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:36.979 INFO:tasks.workunit.client.0.smithi005.stdout: 3 6112 2039.99 8.0 MiB/s 2024-09-13T07:32:37.588 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:37.585+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x562536280400 msgr2=0x7f0180058d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:37.881 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:37.877+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f017c00b6e0 msgr2=0x7f018005b3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:38.068 INFO:tasks.workunit.client.0.smithi005.stdout: 4 7808 1912.07 7.5 MiB/s 2024-09-13T07:32:38.393 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:38.389+0000 7f019d722700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f017c004b90 msgr2=0x7f018003ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:38.491 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:38.489+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f017c004b90 msgr2=0x7f018003ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:38.522 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:38.517+0000 7f019c4c0700 0 -- 172.21.15.5:0/1835909427 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f017c004b90 msgr2=0x7f018003ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:41.653 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 7 ops: 8192 ops/sec: 1066.69 bytes/sec: 4.2 MiB/s 2024-09-13T07:32:41.681 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create rbd/image2@1 2024-09-13T07:32:42.707 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:42.728 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format 2 rbd/image2@1 rbd/test2/image2 2024-09-13T07:32:42.868 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm rbd/image2@1 2024-09-13T07:32:43.001 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:43.020 INFO:tasks.workunit.client.0.smithi005.stderr:+ cmp /dev/fd/63 /dev/fd/62 2024-09-13T07:32:43.020 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export rbd/image2 - 2024-09-13T07:32:43.020 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd export rbd/test2/image2 - 2024-09-13T07:32:43.263 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting imageExporting image: : 44% complete...% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 5% complete... Exporting image: 7% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete...2024-09-13T07:32:43.261+0000 7f023b1a1700 0 -- 172.21.15.5:0/1934167094 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f0218008be0 msgr2=0x7f021800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:43.323 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete...2024-09-13T07:32:43.321+0000 7f023b1a1700 0 -- 172.21.15.5:0/1934167094 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f0220039f40 msgr2=0x7f022003c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:43.550 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image : Exporting image27: % complete...27% complete... Exporting image: 28 % complete...Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 29 % complete...Exporting image: 30% complete... Exporting image: 30% complete... Exporting imageExporting image: : 31% complete...31% complete... Exporting image: 32% complete... Exporting image: 32 % complete...Exporting image: 33% complete... Exporting image : Exporting image33: 34% complete...% complete... Exporting image: 34% complete...2024-09-13T07:32:43.549+0000 7fc08eb08700 0 -- 172.21.15.5:0/997368793 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fc070004be0 msgr2=0x7fc070005040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:43.613 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting imageExporting image: : 3535% complete...% complete... Exporting image: 36Exporting image% complete...: 36% complete... Exporting image: 37Exporting image% complete...: 37% complete... Exporting image : Exporting image38: % complete...38% complete...2024-09-13T07:32:43.609+0000 7fc08eb08700 0 -- 172.21.15.5:0/997368793 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7fc074039f40 msgr2=0x7fc07403c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:44.622 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting imageExporting image: : 4343% complete...% complete... Exporting image : Exporting image44: % complete...44% complete... Exporting imageExporting image: : 4545% complete...% complete... Exporting imageExporting image: : 4646% complete...% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting imageExporting image: : 6363% complete...% complete... Exporting imageExporting image: : 6464% complete...% complete... Exporting imageExporting image: : 6565% complete...% complete... Exporting imageExporting image: : 66% complete...66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting imageExporting image: : 7474% complete...% complete... Exporting image: 75% complete... Exporting image : Exporting image75: % complete...76% complete... Exporting image: Exporting image77: % complete...76% complete... Exporting image: 77% complete... Exporting imageExporting image: : 78% complete...78% complete... Exporting image : Exporting image79: % complete...79% complete... Exporting imageExporting image: : 8080% complete...% complete... Exporting imageExporting image: : 8181% complete...% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting imageExporting image: : 9090% complete...% complete... Exporting imageExporting image: : 9191% complete...% complete... Exporting imageExporting image: : 9292% complete...% complete... Exporting image : Exporting image93: % complete...93% complete... Exporting imageExporting image: : 9494% complete...% complete... Exporting image : Exporting image95: % complete...95% complete... Exporting image: 96Exporting image% complete...: 96% complete... Exporting image: 97Exporting image% complete...: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T07:32:44.623 INFO:tasks.workunit.client.0.smithi005.stderr: Exporting image: 100% complete...done. 2024-09-13T07:32:44.628 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd rm rbd/image2 2024-09-13T07:32:44.628 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd/image2 2024-09-13T07:32:44.716 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:44.713+0000 7f2d01de2780 -1 librbd::api::Image: remove: image has snapshots - not removing 2024-09-13T07:32:44.716 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 0% complete...failed. 2024-09-13T07:32:44.727 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2024-09-13T07:32:44.734 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:44.734 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd/test2/image2 2024-09-13T07:32:45.132 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete...2024-09-13T07:32:45.129+0000 7fd9543d6700 0 -- 172.21.15.5:0/3168573476 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fd938008be0 msgr2=0x7fd93800b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:45.157 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:45.153+0000 7fd9543d6700 0 -- 172.21.15.5:0/3168573476 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fd934039f30 msgr2=0x7fd93403c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:45.758 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:45.764 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd/image2 2024-09-13T07:32:45.974 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete...2024-09-13T07:32:45.969+0000 7f85049d1700 0 -- 172.21.15.5:0/1437231580 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x562906176560 msgr2=0x5629061789f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:45.993 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete...2024-09-13T07:32:45.989+0000 7f85049d1700 0 -- 172.21.15.5:0/1437231580 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f84e403a030 msgr2=0x7f84e403c460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:32:46.378 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T07:32:46.382 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G rbd/test1/image3 2024-09-13T07:32:46.471 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create rbd/test1/image3@1 2024-09-13T07:32:46.718 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:32:46.741 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect rbd/test1/image3@1 2024-09-13T07:32:46.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone --rbd-default-clone-format 1 rbd/test1/image3@1 rbd/test1/image4 2024-09-13T07:32:47.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd/test1/image4 2024-09-13T07:32:47.222 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete...2024-09-13T07:32:47.221+0000 7fa7ac171700 0 -- 172.21.15.5:0/3928639367 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55a4a3e57200 msgr2=0x55a4a3e6b0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:47.273 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:32:47.269+0000 7fa7aa70e700 0 -- 172.21.15.5:0/3928639367 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fa790015a00 msgr2=0x7fa790010810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:47.274 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:47.278 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect rbd/test1/image3@1 2024-09-13T07:32:47.420 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap rm rbd/test1/image3@1 2024-09-13T07:32:47.715 INFO:tasks.workunit.client.0.smithi005.stderr: Removing snap: 100% complete...done. 2024-09-13T07:32:47.738 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd/test1/image3 2024-09-13T07:32:47.939 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete...2024-09-13T07:32:47.937+0000 7f7efab51700 0 -- 172.21.15.5:0/2709601238 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55c7c5f68c00 msgr2=0x55c7c6028370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:47.980 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:47.986 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G --namespace test1 image2 2024-09-13T07:32:48.084 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd namespace remove rbd/test1 2024-09-13T07:32:48.084 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace remove rbd/test1 2024-09-13T07:32:48.144 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: namespace contains images which must be deleted first. 2024-09-13T07:32:48.149 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:48.149 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd group create rbd/test1/group1 2024-09-13T07:32:48.210 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd group image add rbd/test1/group1 rbd/test1/image1 2024-09-13T07:32:48.290 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd group rm rbd/test1/group1 2024-09-13T07:32:48.370 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash move rbd/test1/image1 2024-09-13T07:32:48.494 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash --namespace test1 ls 2024-09-13T07:32:48.494 INFO:tasks.workunit.client.0.smithi005.stderr:++ cut -d ' ' -f 1 2024-09-13T07:32:48.547 INFO:tasks.workunit.client.0.smithi005.stderr:+ ID=29a18256b5e7 2024-09-13T07:32:48.547 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash rm rbd/test1/29a18256b5e7 2024-09-13T07:32:48.762 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete...2024-09-13T07:32:48.757+0000 7fd95f03a700 0 -- 172.21.15.5:0/498373069 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55e7bb9c8160 msgr2=0x55e7bb9dea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:48.788 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete...2024-09-13T07:32:48.785+0000 7fd95f03a700 0 -- 172.21.15.5:0/498373069 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fd94003a050 msgr2=0x7fd94003c480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:49.074 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T07:32:49.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd remove rbd/test1/image2 2024-09-13T07:32:49.273 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete...2024-09-13T07:32:49.269+0000 7fc7dbf05700 0 -- 172.21.15.5:0/2636484906 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x55cd807852d0 msgr2=0x55cd8079b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:32:49.311 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:32:49.317 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace remove --pool rbd --namespace test1 2024-09-13T07:32:49.419 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace remove --namespace test3 2024-09-13T07:32:49.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace list 2024-09-13T07:32:49.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep test 2024-09-13T07:32:49.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:32:49.525 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:32:49.575 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:32:49.575 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace remove rbd/test2 2024-09-13T07:32:49.674 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_trash_purge_schedule 2024-09-13T07:32:49.674 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing trash purge schedule...' 2024-09-13T07:32:49.674 INFO:tasks.workunit.client.0.smithi005.stdout:testing trash purge schedule... 2024-09-13T07:32:49.674 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:32:49.675 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:49.830 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:49.992 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:50.139 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:50.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:50.448 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:50.606 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:50.759 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:50.925 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:51.083 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:51.236 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:51.393 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:51.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:51.705 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:51.863 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:52.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:52.390 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:52.550 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:32:52.705 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:32:53.822 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:32:53.831 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:32:56.691 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd2/ns1 2024-09-13T07:32:56.754 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd trash purge schedule list 2024-09-13T07:32:57.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '{}' = '{}' 2024-09-13T07:32:57.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd trash purge schedule status 2024-09-13T07:32:57.256 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep '"scheduled": []' 2024-09-13T07:32:57.748 INFO:tasks.workunit.client.0.smithi005.stdout: "scheduled": [] 2024-09-13T07:32:57.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule ls 2024-09-13T07:32:57.749 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:32:57.794 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:57.795 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -R --format json 2024-09-13T07:32:57.842 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:32:57.843 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd 1d 01:30 2024-09-13T07:32:57.901 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd 2024-09-13T07:32:57.901 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1d starting at 01:30' 2024-09-13T07:32:57.955 INFO:tasks.workunit.client.0.smithi005.stdout:every 1d starting at 01:30:00 2024-09-13T07:32:57.955 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule ls 2024-09-13T07:32:57.955 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:32:58.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:58.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -R 2024-09-13T07:32:58.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1d starting at 01:30' 2024-09-13T07:32:58.057 INFO:tasks.workunit.client.0.smithi005.stdout:rbd - every 1d starting at 01:30:00 2024-09-13T07:32:58.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -R -p rbd 2024-09-13T07:32:58.058 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1d starting at 01:30' 2024-09-13T07:32:58.112 INFO:tasks.workunit.client.0.smithi005.stdout:rbd - every 1d starting at 01:30:00 2024-09-13T07:32:58.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule ls -p rbd2 2024-09-13T07:32:58.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd2 2024-09-13T07:32:58.167 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:32:58.168 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -p rbd2 -R --format json 2024-09-13T07:32:58.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:32:58.225 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd2/ns1 2d 2024-09-13T07:32:58.289 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -p rbd2 -R --format json 2024-09-13T07:32:58.350 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[{"pool":"rbd2","namespace":"ns1","items":[{"interval":"2d","start_time":""}]}]' '!=' '[]' 2024-09-13T07:32:58.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd2 -R 2024-09-13T07:32:58.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *every 2d' 2024-09-13T07:32:58.414 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 every 2d 2024-09-13T07:32:58.414 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule rm -p rbd2/ns1 2024-09-13T07:32:58.477 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -p rbd2 -R --format json 2024-09-13T07:32:58.536 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:32:58.536 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:32:58.538 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:32:58.540 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:32:58.540 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:32:58.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '' = rbd 2024-09-13T07:32:58.597 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:33:08.599 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:08.600 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:33:08.600 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '' = rbd 2024-09-13T07:33:08.661 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:33:18.662 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:18.664 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:33:18.664 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:18.722 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '' = rbd 2024-09-13T07:33:18.722 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:33:28.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:28.725 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:33:28.725 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:28.784 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '' = rbd 2024-09-13T07:33:28.784 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:33:38.785 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:38.787 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:33:38.787 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:38.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd = rbd 2024-09-13T07:33:38.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:33:38.849 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status 2024-09-13T07:33:38.895 INFO:tasks.workunit.client.0.smithi005.stdout:POOL NAMESPACE SCHEDULE TIME 2024-09-13T07:33:38.895 INFO:tasks.workunit.client.0.smithi005.stdout:rbd 2024-09-14 01:30:00 2024-09-13T07:33:38.903 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:33:38.903 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:38.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd = rbd 2024-09-13T07:33:38.959 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status -p rbd --format xml 2024-09-13T07:33:38.959 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:39.013 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd = rbd 2024-09-13T07:33:39.013 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add 2d 00:17 2024-09-13T07:33:39.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:33:39.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2d starting at 00:17' 2024-09-13T07:33:39.133 INFO:tasks.workunit.client.0.smithi005.stdout:every 2d starting at 00:17:00 2024-09-13T07:33:39.133 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -R 2024-09-13T07:33:39.134 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2d starting at 00:17' 2024-09-13T07:33:39.188 INFO:tasks.workunit.client.0.smithi005.stdout:- - every 2d starting at 00:17:00 2024-09-13T07:33:39.189 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule ls -p rbd2 2024-09-13T07:33:39.189 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd2 2024-09-13T07:33:39.244 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:33:39.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd2 -R 2024-09-13T07:33:39.245 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2d starting at 00:17' 2024-09-13T07:33:39.300 INFO:tasks.workunit.client.0.smithi005.stdout:- - every 2d starting at 00:17:00 2024-09-13T07:33:39.301 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd2/ns1 -R 2024-09-13T07:33:39.301 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2d starting at 00:17' 2024-09-13T07:33:39.360 INFO:tasks.workunit.client.0.smithi005.stdout:- - every 2d starting at 00:17:00 2024-09-13T07:33:39.361 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -R -p rbd2/ns1 --format xml 2024-09-13T07:33:39.362 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //schedules/schedule/pool 2024-09-13T07:33:39.426 INFO:tasks.workunit.client.0.smithi005.stderr:+ test - = - 2024-09-13T07:33:39.427 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -R -p rbd2/ns1 --format xml 2024-09-13T07:33:39.427 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //schedules/schedule/namespace 2024-09-13T07:33:39.489 INFO:tasks.workunit.client.0.smithi005.stderr:+ test - = - 2024-09-13T07:33:39.490 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -R -p rbd2/ns1 --format xml 2024-09-13T07:33:39.490 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //schedules/schedule/items/item/start_time 2024-09-13T07:33:39.551 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 00:17:00 = 00:17:00 2024-09-13T07:33:39.551 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:33:39.552 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:39.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:33:39.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:39.553 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:33:39.608 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:33:49.609 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:49.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:33:49.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:49.610 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:33:49.666 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:33:59.667 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:33:59.668 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:33:59.668 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:33:59.668 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:33:59.725 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:34:09.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:34:09.727 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:34:09.728 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:09.728 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:34:09.784 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:34:19.786 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:34:19.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:34:19.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:19.787 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:34:19.848 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:34:29.850 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:34:29.850 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:34:29.851 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:29.851 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:34:29.909 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:34:39.910 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:34:39.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:34:39.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:39.912 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:34:39.971 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 2024-09-13T07:34:39.971 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 2024-09-13T07:34:39.971 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:34:39.972 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status 2024-09-13T07:34:40.017 INFO:tasks.workunit.client.0.smithi005.stdout:POOL NAMESPACE SCHEDULE TIME 2024-09-13T07:34:40.017 INFO:tasks.workunit.client.0.smithi005.stdout:datapool 2024-09-14 00:17:00 2024-09-13T07:34:40.017 INFO:tasks.workunit.client.0.smithi005.stdout:rbd 2024-09-14 01:30:00 2024-09-13T07:34:40.017 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 2024-09-14 00:17:00 2024-09-13T07:34:40.017 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-14 00:17:00 2024-09-13T07:34:40.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status --format xml 2024-09-13T07:34:40.021 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:40.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2 2024-09-13T07:34:40.073 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 2024-09-13T07:34:40.074 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 2024-09-13T07:34:40.074 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd rbd2 rbd2' 2024-09-13T07:34:40.075 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status --format xml 2024-09-13T07:34:40.076 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:40.133 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo datapool rbd rbd2 rbd2 2024-09-13T07:34:40.133 INFO:tasks.workunit.client.0.smithi005.stdout:datapool rbd rbd2 rbd2 2024-09-13T07:34:40.134 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule status -p rbd --format xml 2024-09-13T07:34:40.135 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:40.191 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd = rbd 2024-09-13T07:34:40.193 INFO:tasks.workunit.client.0.smithi005.stderr:+++ rbd trash purge schedule status -p rbd2 --format xml 2024-09-13T07:34:40.193 INFO:tasks.workunit.client.0.smithi005.stderr:+++ xmlstarlet sel -t -v //scheduled/item/pool 2024-09-13T07:34:40.254 INFO:tasks.workunit.client.0.smithi005.stderr:++ echo rbd2 rbd2 2024-09-13T07:34:40.255 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'rbd2 rbd2' = 'rbd2 rbd2' 2024-09-13T07:34:40.257 INFO:tasks.workunit.client.0.smithi005.stderr:+++ rbd trash purge schedule ls -R --format xml 2024-09-13T07:34:40.257 INFO:tasks.workunit.client.0.smithi005.stderr:+++ xmlstarlet sel -t -v //schedules/schedule/items 2024-09-13T07:34:40.317 INFO:tasks.workunit.client.0.smithi005.stderr:++ echo 2d00:17:00 1d01:30:00 2024-09-13T07:34:40.317 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '2d00:17:00 1d01:30:00' = '2d00:17:00 1d01:30:00' 2024-09-13T07:34:40.317 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add 1d 2024-09-13T07:34:40.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:34:40.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2d starting at 00:17' 2024-09-13T07:34:40.441 INFO:tasks.workunit.client.0.smithi005.stdout:every 1d, every 2d starting at 00:17:00 2024-09-13T07:34:40.441 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:34:40.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1d' 2024-09-13T07:34:40.492 INFO:tasks.workunit.client.0.smithi005.stdout:every 1d, every 2d starting at 00:17:00 2024-09-13T07:34:40.493 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -R --format xml 2024-09-13T07:34:40.493 INFO:tasks.workunit.client.0.smithi005.stderr:+ xmlstarlet sel -t -v //schedules/schedule/items 2024-09-13T07:34:40.493 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 2d00:17 2024-09-13T07:34:40.549 INFO:tasks.workunit.client.0.smithi005.stdout:1d2d00:17:00 2024-09-13T07:34:40.550 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule rm 1d 2024-09-13T07:34:40.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:34:40.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2d starting at 00:17' 2024-09-13T07:34:40.665 INFO:tasks.workunit.client.0.smithi005.stdout:every 2d starting at 00:17:00 2024-09-13T07:34:40.666 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule rm 2d 00:17 2024-09-13T07:34:40.731 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule ls 2024-09-13T07:34:40.731 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:34:40.776 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:34:40.776 INFO:tasks.workunit.client.0.smithi005.stderr:+ for p in rbd2 rbd2/ns1 2024-09-13T07:34:40.776 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 rbd2/ns1/test1 2024-09-13T07:34:40.859 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv rbd2/ns1/test1 2024-09-13T07:34:40.954 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:34:40.955 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:34:40.955 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:34:41.005 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:34:41.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd2 1m 2024-09-13T07:34:41.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2 -R 2024-09-13T07:34:41.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:34:41.126 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 - every 1m 2024-09-13T07:34:41.127 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:34:41.127 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2/ns1 -R 2024-09-13T07:34:41.183 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 - every 1m 2024-09-13T07:34:41.184 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:34:41.185 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:34:41.186 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:34:41.186 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:34:41.187 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:34:41.233 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:34:41.233 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:34:51.235 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:34:51.236 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:34:51.236 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:34:51.236 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:34:51.288 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:34:51.288 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:35:01.290 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:01.291 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:01.291 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:01.291 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:01.345 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:35:01.345 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:01.346 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:01.346 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:35:01.396 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:35:01.397 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2 -R 2024-09-13T07:35:01.397 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:35:01.453 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 - every 1m 2024-09-13T07:35:01.453 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2/ns1 -R 2024-09-13T07:35:01.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:35:01.511 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 - every 1m 2024-09-13T07:35:01.512 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status 2024-09-13T07:35:01.512 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1' 2024-09-13T07:35:01.564 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-13 07:36:00 2024-09-13T07:35:01.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status -p rbd2 2024-09-13T07:35:01.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1' 2024-09-13T07:35:01.615 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-13 07:36:00 2024-09-13T07:35:01.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1' 2024-09-13T07:35:01.617 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status -p rbd2/ns1 2024-09-13T07:35:01.672 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-13 07:36:00 2024-09-13T07:35:01.672 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule rm -p rbd2 1m 2024-09-13T07:35:01.731 INFO:tasks.workunit.client.0.smithi005.stderr:+ for p in rbd2 rbd2/ns1 2024-09-13T07:35:01.732 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 rbd2/ns1/test1 2024-09-13T07:35:01.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash mv rbd2/ns1/test1 2024-09-13T07:35:01.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:01.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:01.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:01.961 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:01.961 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd2/ns1 1m 2024-09-13T07:35:02.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2 -R 2024-09-13T07:35:02.022 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:35:02.078 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 every 1m 2024-09-13T07:35:02.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2/ns1 -R 2024-09-13T07:35:02.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:35:02.132 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 every 1m 2024-09-13T07:35:02.133 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:35:02.134 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:02.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:02.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:02.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:02.184 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:02.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:35:12.187 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:12.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:12.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:12.188 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:12.240 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:12.241 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:35:22.242 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:22.243 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:22.243 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:22.243 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:22.294 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:22.294 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:35:32.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:32.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:32.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:32.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:32.350 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:32.350 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:35:42.352 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:42.353 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:42.353 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:42.353 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:42.406 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:42.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:35:52.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:35:52.408 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:35:52.409 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:35:52.409 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:35:52.466 INFO:tasks.workunit.client.0.smithi005.stdout:1 2024-09-13T07:35:52.467 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:36:02.468 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:36:02.469 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:36:02.469 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:36:02.469 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^1$' 2024-09-13T07:36:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:36:02.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash ls rbd2/ns1 2024-09-13T07:36:02.525 INFO:tasks.workunit.client.0.smithi005.stderr:+ wc -l 2024-09-13T07:36:02.525 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep '^0$' 2024-09-13T07:36:02.575 INFO:tasks.workunit.client.0.smithi005.stdout:0 2024-09-13T07:36:02.575 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2 -R 2024-09-13T07:36:02.576 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:36:02.632 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 every 1m 2024-09-13T07:36:02.633 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule list -p rbd2/ns1 -R 2024-09-13T07:36:02.633 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:36:02.687 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 every 1m 2024-09-13T07:36:02.688 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status 2024-09-13T07:36:02.688 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1' 2024-09-13T07:36:02.738 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-13 07:37:00 2024-09-13T07:36:02.739 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status -p rbd2 2024-09-13T07:36:02.739 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1' 2024-09-13T07:36:02.794 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-13 07:37:00 2024-09-13T07:36:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule status -p rbd2/ns1 2024-09-13T07:36:02.795 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1' 2024-09-13T07:36:02.854 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 2024-09-13 07:37:00 2024-09-13T07:36:02.855 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule rm -p rbd2/ns1 1m 2024-09-13T07:36:02.917 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add 2m 2024-09-13T07:36:02.983 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule add -p rbd dummy 2024-09-13T07:36:02.983 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd dummy 2024-09-13T07:36:03.034 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:03.032+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:36:03.035 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule add failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:36:03.040 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:03.040 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule add dummy 2024-09-13T07:36:03.041 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add dummy 2024-09-13T07:36:03.086 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:03.080+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:36:03.086 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule add failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:36:03.091 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:03.091 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule remove -p rbd dummy 2024-09-13T07:36:03.091 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule remove -p rbd dummy 2024-09-13T07:36:03.141 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:03.136+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:36:03.141 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule remove failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:36:03.145 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:03.145 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule remove dummy 2024-09-13T07:36:03.145 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule remove dummy 2024-09-13T07:36:03.194 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:03.192+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:36:03.195 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule remove failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:36:03.200 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:03.200 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd 2024-09-13T07:36:03.200 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1d starting at 01:30' 2024-09-13T07:36:03.260 INFO:tasks.workunit.client.0.smithi005.stdout:every 1d starting at 01:30:00 2024-09-13T07:36:03.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls 2024-09-13T07:36:03.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2m' 2024-09-13T07:36:03.315 INFO:tasks.workunit.client.0.smithi005.stdout:every 2m 2024-09-13T07:36:03.315 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule remove -p rbd 1d 01:30 2024-09-13T07:36:03.380 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule remove 2m 2024-09-13T07:36:03.445 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd trash purge schedule ls -R --format json 2024-09-13T07:36:03.500 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:36:03.501 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:36:03.501 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:03.652 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:03.807 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:03.964 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:04.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:04.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:04.438 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:04.596 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:04.754 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:04.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:05.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:05.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:05.589 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:05.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:05.907 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:06.056 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:06.220 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:06.375 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:06.537 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:36:07.584 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:36:07.594 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_trash_purge_schedule_recovery 2024-09-13T07:36:07.594 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing recovery of trash_purge_schedule handler after module'\''s RADOS client is blocklisted...' 2024-09-13T07:36:07.594 INFO:tasks.workunit.client.0.smithi005.stdout:testing recovery of trash_purge_schedule handler after module's RADOS client is blocklisted... 2024-09-13T07:36:07.595 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:36:07.595 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:07.739 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:07.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:08.053 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:08.220 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:08.379 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:08.546 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:08.705 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:08.857 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.174 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.330 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.490 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.645 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.810 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:09.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:10.131 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:10.292 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:10.452 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd3 8 2024-09-13T07:36:11.600 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd3' already exists 2024-09-13T07:36:11.609 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd3 2024-09-13T07:36:14.524 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd3/ns1 2024-09-13T07:36:14.578 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd3/ns1 2d 2024-09-13T07:36:14.639 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd3 -R 2024-09-13T07:36:14.640 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd3 *ns1 *every 2d' 2024-09-13T07:36:14.694 INFO:tasks.workunit.client.0.smithi005.stdout:rbd3 ns1 every 2d 2024-09-13T07:36:14.695 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph mgr dump 2024-09-13T07:36:14.696 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-13T07:36:14.696 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq '.active_clients[]' 2024-09-13T07:36:14.696 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-13T07:36:15.205 INFO:tasks.workunit.client.0.smithi005.stderr:+ CLIENT_ADDR=172.21.15.5:0/3795888129 2024-09-13T07:36:15.206 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd blocklist add 172.21.15.5:0/3795888129 2024-09-13T07:36:17.526 INFO:tasks.workunit.client.0.smithi005.stderr:blocklisting 172.21.15.5:0/3795888129 until 2024-09-13T08:36:16.637784+0000 (3600 sec) 2024-09-13T07:36:17.537 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd trash purge schedule add -p rbd3 10m 2024-09-13T07:36:17.537 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd3 10m 2024-09-13T07:36:17.592 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:17.588+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable [errno 108] RADOS connection was shutdown (Failed to operate write op for oid rbd_trash_purge_schedule) 2024-09-13T07:36:17.592 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule add failed: (11) Resource temporarily unavailable: [errno 108] RADOS connection was shutdown (Failed to operate write op for oid rbd_trash_purge_schedule) 2024-09-13T07:36:17.596 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:17.596 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:36:27.599 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 24 2024-09-13T07:36:27.600 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:36:27.600 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd3 10m 2024-09-13T07:36:27.650 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:27.644+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:36:27.651 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:36:27.656 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:36:37.658 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:36:37.658 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd3 10m 2024-09-13T07:36:37.711 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:36:37.708+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:36:37.711 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd trash purge schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:36:37.717 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:36:47.719 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:36:47.719 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule add -p rbd3 10m 2024-09-13T07:36:47.784 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:36:47.784 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd3 -R 2024-09-13T07:36:47.785 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 10m' 2024-09-13T07:36:47.841 INFO:tasks.workunit.client.0.smithi005.stdout:rbd3 - every 10m 2024-09-13T07:36:47.842 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd3 -R 2024-09-13T07:36:47.842 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd3 *ns1 *every 2d' 2024-09-13T07:36:47.897 INFO:tasks.workunit.client.0.smithi005.stdout:rbd3 ns1 every 2d 2024-09-13T07:36:47.897 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule remove -p rbd3 10m 2024-09-13T07:36:47.960 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule remove -p rbd3/ns1 2d 2024-09-13T07:36:48.023 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd3 -R 2024-09-13T07:36:48.024 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep 'every 10m' 2024-09-13T07:36:48.024 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 10m' 2024-09-13T07:36:48.080 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:48.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd trash purge schedule ls -p rbd3 -R 2024-09-13T07:36:48.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep 'rbd3 *ns1 *every 2d' 2024-09-13T07:36:48.081 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd3 *ns1 *every 2d' 2024-09-13T07:36:48.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:48.135 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd3 rbd3 --yes-i-really-really-mean-it 2024-09-13T07:36:48.885 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd3' does not exist 2024-09-13T07:36:48.894 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_mirror_snapshot_schedule 2024-09-13T07:36:48.894 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing mirror snapshot schedule...' 2024-09-13T07:36:48.894 INFO:tasks.workunit.client.0.smithi005.stdout:testing mirror snapshot schedule... 2024-09-13T07:36:48.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:36:48.895 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.032 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.187 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.351 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.517 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.678 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.832 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:49.991 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:50.155 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:50.309 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:50.470 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:50.627 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:50.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:50.953 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:51.116 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:51.274 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:51.429 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:51.803 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:36:51.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:36:52.892 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:36:52.901 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:36:55.802 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd2/ns1 2024-09-13T07:36:55.861 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool enable rbd2 image 2024-09-13T07:36:55.918 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool enable rbd2/ns1 image 2024-09-13T07:36:55.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool peer add rbd2 cluster1 2024-09-13T07:36:56.031 INFO:tasks.workunit.client.0.smithi005.stdout:67b4d4f0-11aa-427d-b996-0558c76f17bd 2024-09-13T07:36:56.037 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd mirror snapshot schedule list 2024-09-13T07:36:56.547 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '{}' = '{}' 2024-09-13T07:36:56.548 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd mirror snapshot schedule status 2024-09-13T07:36:56.548 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep '"scheduled_images": []' 2024-09-13T07:36:57.041 INFO:tasks.workunit.client.0.smithi005.stdout: "scheduled_images": [] 2024-09-13T07:36:57.042 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls 2024-09-13T07:36:57.042 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls 2024-09-13T07:36:57.093 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:57.093 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -R --format json 2024-09-13T07:36:57.144 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:36:57.144 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 rbd2/ns1/test1 2024-09-13T07:36:57.223 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:36:57.223 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:36:57.287 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: mirroring not enabled on the image 2024-09-13T07:36:57.295 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 0 = 0 2024-09-13T07:36:57.295 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror image enable rbd2/ns1/test1 snapshot 2024-09-13T07:36:57.840 INFO:tasks.workunit.client.0.smithi005.stdout:Mirroring enabled 2024-09-13T07:36:57.857 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:36:57.857 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:36:57.930 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 = 1 2024-09-13T07:36:57.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd2/ns1 --image test1 1m 2024-09-13T07:36:58.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls 2024-09-13T07:36:58.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls 2024-09-13T07:36:58.065 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:58.066 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:36:58.066 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -R 2024-09-13T07:36:58.124 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:36:58.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls -p rbd2 2024-09-13T07:36:58.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 2024-09-13T07:36:58.181 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:58.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 -R 2024-09-13T07:36:58.182 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:36:58.239 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:36:58.240 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls -p rbd2/ns1 2024-09-13T07:36:58.240 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 2024-09-13T07:36:58.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:36:58.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 -R 2024-09-13T07:36:58.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:36:58.356 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:36:58.358 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -p rbd2/ns1 --image test1 2024-09-13T07:36:58.437 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1m' = 'every 1m' 2024-09-13T07:36:58.437 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:36:58.438 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:36:58.439 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:36:58.440 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:36:58.521 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:36:58.521 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:37:08.522 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:37:08.523 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:37:08.524 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:37:08.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:37:08.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:37:18.605 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:37:18.606 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:37:18.606 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:37:18.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:37:18.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:37:28.686 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:37:28.687 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:37:28.688 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:37:28.764 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:37:28.765 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:37:38.766 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:37:38.767 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:37:38.767 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:37:38.844 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:37:38.845 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:37:48.846 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:37:48.847 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:37:48.848 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:37:48.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:37:48.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:37:58.928 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:37:58.932 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:37:58.933 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:37:59.005 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 -gt 1 2024-09-13T07:37:59.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:38:09.007 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:09.009 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:38:09.009 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:38:09.087 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 2 -gt 1 2024-09-13T07:38:09.087 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:38:09.088 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd2/ns1/test1 2024-09-13T07:38:09.088 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:38:09.165 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 2 -gt 1 2024-09-13T07:38:09.165 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls 2024-09-13T07:38:09.166 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls 2024-09-13T07:38:09.218 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:38:09.218 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -R 2024-09-13T07:38:09.218 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:38:09.273 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:38:09.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls -p rbd2 2024-09-13T07:38:09.273 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 2024-09-13T07:38:09.329 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:38:09.329 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 -R 2024-09-13T07:38:09.330 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:38:09.384 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:38:09.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls -p rbd2/ns1 2024-09-13T07:38:09.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 2024-09-13T07:38:09.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:38:09.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 -R 2024-09-13T07:38:09.443 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:38:09.502 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:38:09.503 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -p rbd2/ns1 --image test1 2024-09-13T07:38:09.577 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1m' = 'every 1m' 2024-09-13T07:38:09.577 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:09.624 INFO:tasks.workunit.client.0.smithi005.stdout:SCHEDULE TIME IMAGE 2024-09-13T07:38:09.624 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:39:00 rbd2/ns1/test1 2024-09-13T07:38:09.629 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule status --format xml 2024-09-13T07:38:09.629 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled_images/image/image 2024-09-13T07:38:09.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd2/ns1/test1 = rbd2/ns1/test1 2024-09-13T07:38:09.685 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule status -p rbd2 --format xml 2024-09-13T07:38:09.686 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled_images/image/image 2024-09-13T07:38:09.739 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd2/ns1/test1 = rbd2/ns1/test1 2024-09-13T07:38:09.740 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule status -p rbd2/ns1 --format xml 2024-09-13T07:38:09.741 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled_images/image/image 2024-09-13T07:38:09.798 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd2/ns1/test1 = rbd2/ns1/test1 2024-09-13T07:38:09.799 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule status -p rbd2/ns1 --image test1 --format xml 2024-09-13T07:38:09.799 INFO:tasks.workunit.client.0.smithi005.stderr:++ xmlstarlet sel -t -v //scheduled_images/image/image 2024-09-13T07:38:09.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ test rbd2/ns1/test1 = rbd2/ns1/test1 2024-09-13T07:38:09.871 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror image demote rbd2/ns1/test1 2024-09-13T07:38:10.887 INFO:tasks.workunit.client.0.smithi005.stdout:Image demoted to non-primary 2024-09-13T07:38:10.898 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:38:10.899 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:10.900 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:10.900 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:10.950 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:39:00 rbd2/ns1/test1 2024-09-13T07:38:10.951 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:38:20.952 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:20.953 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:20.953 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:21.008 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:39:00 rbd2/ns1/test1 2024-09-13T07:38:21.009 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:38:31.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:31.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:31.011 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:31.069 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:39:00 rbd2/ns1/test1 2024-09-13T07:38:31.069 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:38:41.071 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:41.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:41.072 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:41.126 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:39:00 rbd2/ns1/test1 2024-09-13T07:38:41.126 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:38:51.128 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:51.128 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:51.129 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:51.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:38:51.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:51.185 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep rbd2/ns1/test1 2024-09-13T07:38:51.185 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:51.237 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:38:51.237 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror image promote rbd2/ns1/test1 2024-09-13T07:38:51.515 INFO:tasks.workunit.client.0.smithi005.stdout:Image promoted to primary 2024-09-13T07:38:51.531 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:38:51.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:38:51.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:38:51.533 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:38:51.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:39:01.587 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:01.588 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:01.588 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:01.642 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:39:11.644 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:11.645 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:11.645 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:11.700 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:39:21.702 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:21.702 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:21.702 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:21.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:39:31.764 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:31.765 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:31.765 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:31.822 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:39:41.824 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:41.825 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:41.825 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:41.879 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:40:00 rbd2/ns1/test1 2024-09-13T07:39:41.879 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:39:41.880 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:41.880 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:41.931 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:40:00 rbd2/ns1/test1 2024-09-13T07:39:41.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add 1h 00:15 2024-09-13T07:39:42.002 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls 2024-09-13T07:39:42.054 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1h starting at 00:15:00' = 'every 1h starting at 00:15:00' 2024-09-13T07:39:42.055 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -R 2024-09-13T07:39:42.055 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1h starting at 00:15:00' 2024-09-13T07:39:42.112 INFO:tasks.workunit.client.0.smithi005.stdout:- - - every 1h starting at 00:15:00 2024-09-13T07:39:42.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -R 2024-09-13T07:39:42.113 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:39:42.165 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:39:42.165 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls -p rbd2 2024-09-13T07:39:42.166 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 2024-09-13T07:39:42.221 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:39:42.222 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 -R 2024-09-13T07:39:42.222 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1h starting at 00:15:00' 2024-09-13T07:39:42.283 INFO:tasks.workunit.client.0.smithi005.stdout:- - - every 1h starting at 00:15:00 2024-09-13T07:39:42.284 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2 -R 2024-09-13T07:39:42.284 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:39:42.341 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:39:42.341 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule ls -p rbd2/ns1 2024-09-13T07:39:42.342 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 2024-09-13T07:39:42.397 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:39:42.398 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 -R 2024-09-13T07:39:42.398 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1h starting at 00:15:00' 2024-09-13T07:39:42.457 INFO:tasks.workunit.client.0.smithi005.stdout:- - - every 1h starting at 00:15:00 2024-09-13T07:39:42.457 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd2/ns1 -R 2024-09-13T07:39:42.457 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'rbd2 *ns1 *test1 *every 1m' 2024-09-13T07:39:42.514 INFO:tasks.workunit.client.0.smithi005.stdout:rbd2 ns1 test1 every 1m 2024-09-13T07:39:42.515 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -p rbd2/ns1 --image test1 2024-09-13T07:39:42.588 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1m' = 'every 1m' 2024-09-13T07:39:42.588 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule add dummy 2024-09-13T07:39:42.588 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add dummy 2024-09-13T07:39:42.637 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:39:42.632+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:39:42.637 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:39:42.643 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:39:42.643 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule add -p rbd2/ns1 --image test1 dummy 2024-09-13T07:39:42.644 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd2/ns1 --image test1 dummy 2024-09-13T07:39:42.715 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:39:42.712+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:39:42.716 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:39:42.722 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:39:42.722 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule remove dummy 2024-09-13T07:39:42.722 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule remove dummy 2024-09-13T07:39:42.773 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:39:42.768+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:39:42.773 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule remove failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:39:42.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:39:42.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule remove -p rbd2/ns1 --image test1 dummy 2024-09-13T07:39:42.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule remove -p rbd2/ns1 --image test1 dummy 2024-09-13T07:39:42.851 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:39:42.848+0000 7fbaa71df700 -1 mgr.server reply reply (22) Invalid argument Invalid interval (dummy) 2024-09-13T07:39:42.852 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule remove failed: (22) Invalid argument: Invalid interval (dummy) 2024-09-13T07:39:42.857 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:39:42.858 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls 2024-09-13T07:39:42.911 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1h starting at 00:15:00' = 'every 1h starting at 00:15:00' 2024-09-13T07:39:42.911 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -p rbd2/ns1 --image test1 2024-09-13T07:39:42.988 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1m' = 'every 1m' 2024-09-13T07:39:42.988 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd2/ns1/test1 2024-09-13T07:39:44.015 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:39:44.012+0000 7fa6eb841700 0 -- 172.21.15.5:0/3448132720 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x56532a7b43f0 msgr2=0x56532a7ca560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:39:45.021 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:39:45.015+0000 7fa6eb841700 0 -- 172.21.15.5:0/3448132720 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fa6cc03abe0 msgr2=0x7fa6cc03cfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:39:47.112 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:39:47.117 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 12 2024-09-13T07:39:47.118 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:47.119 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:47.119 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:47.170 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:40:00 rbd2/ns1/test1 2024-09-13T07:39:47.170 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:39:57.172 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:39:57.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:39:57.173 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:39:57.230 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:40:00 rbd2/ns1/test1 2024-09-13T07:39:57.231 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:40:07.233 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:40:07.233 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:40:07.234 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:40:07.287 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:41:00 rbd2/ns1/test1 2024-09-13T07:40:07.288 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:40:17.290 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:40:17.290 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:40:17.290 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:40:17.347 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:41:00 rbd2/ns1/test1 2024-09-13T07:40:17.347 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:40:27.349 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:40:27.350 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:40:27.350 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:40:27.402 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:41:00 rbd2/ns1/test1 2024-09-13T07:40:27.402 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:40:37.404 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:40:37.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:40:37.405 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:40:37.460 INFO:tasks.workunit.client.0.smithi005.stdout:2024-09-13 07:41:00 rbd2/ns1/test1 2024-09-13T07:40:37.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:40:47.462 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 12` 2024-09-13T07:40:47.463 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:40:47.463 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:40:47.518 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:40:47.518 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule status 2024-09-13T07:40:47.518 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep rbd2/ns1/test1 2024-09-13T07:40:47.519 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep rbd2/ns1/test1 2024-09-13T07:40:47.569 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:40:47.569 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule remove 2024-09-13T07:40:47.635 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -R --format json 2024-09-13T07:40:47.694 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:40:47.694 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:40:47.694 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:47.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:48.217 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:48.362 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:48.517 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:48.674 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:48.839 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.164 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.323 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.485 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.645 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.807 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:49.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:50.119 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:50.277 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:50.437 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:50.603 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:50.765 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:40:51.935 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:40:51.944 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_mirror_snapshot_schedule_recovery 2024-09-13T07:40:51.944 INFO:tasks.workunit.client.0.smithi005.stdout:testing recovery of mirror snapshot scheduler after module's RADOS client is blocklisted... 2024-09-13T07:40:51.945 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing recovery of mirror snapshot scheduler after module'\''s RADOS client is blocklisted...' 2024-09-13T07:40:51.945 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:40:51.945 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:52.089 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:52.250 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:52.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:52.568 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:52.730 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:52.894 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:53.052 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:53.209 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:53.372 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:53.531 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:53.692 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:54.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:54.221 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:54.384 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:54.541 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:54.702 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:54.860 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:40:55.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd3 8 2024-09-13T07:40:55.960 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd3' already exists 2024-09-13T07:40:55.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd3 2024-09-13T07:40:58.874 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd3/ns1 2024-09-13T07:40:58.930 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool enable rbd3 image 2024-09-13T07:40:58.986 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool enable rbd3/ns1 image 2024-09-13T07:40:59.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool peer add rbd3 cluster1 2024-09-13T07:40:59.096 INFO:tasks.workunit.client.0.smithi005.stdout:8e5542b7-02ac-493e-a22e-b0531c09aafc 2024-09-13T07:40:59.100 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 -s 1 rbd3/ns1/test1 2024-09-13T07:40:59.184 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror image enable rbd3/ns1/test1 snapshot 2024-09-13T07:40:59.895 INFO:tasks.workunit.client.0.smithi005.stdout:Mirroring enabled 2024-09-13T07:40:59.914 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror image status rbd3/ns1/test1 2024-09-13T07:40:59.914 INFO:tasks.workunit.client.0.smithi005.stderr:++ grep -c mirror.primary 2024-09-13T07:40:59.994 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 1 = 1 2024-09-13T07:40:59.994 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 1m 2024-09-13T07:41:00.074 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror snapshot schedule ls -p rbd3/ns1 --image test1 2024-09-13T07:41:00.152 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'every 1m' = 'every 1m' 2024-09-13T07:41:00.153 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph mgr dump 2024-09-13T07:41:00.154 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-13T07:41:00.154 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq '.active_clients[]' 2024-09-13T07:41:00.155 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-13T07:41:00.659 INFO:tasks.workunit.client.0.smithi005.stderr:+ CLIENT_ADDR=172.21.15.5:0/2170969492 2024-09-13T07:41:00.660 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd blocklist add 172.21.15.5:0/2170969492 2024-09-13T07:41:02.894 INFO:tasks.workunit.client.0.smithi005.stderr:blocklisting 172.21.15.5:0/2170969492 until 2024-09-13T08:41:01.999929+0000 (3600 sec) 2024-09-13T07:41:02.906 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:02.907 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:02.958 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:41:02.958+0000 7fbaa71df700 -1 librbd::api::Namespace: list: error listing namespaces: (108) Cannot send after transport endpoint shutdown 2024-09-13T07:41:02.959 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:41:02.958+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable [errno 108] RBD connection was shutdown (error listing namespaces) 2024-09-13T07:41:02.959 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: [errno 108] RBD connection was shutdown (error listing namespaces) 2024-09-13T07:41:02.964 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:41:02.964 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:41:12.966 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 24 2024-09-13T07:41:12.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:41:12.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:13.021 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:41:13.018+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:41:13.022 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:41:13.025 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:41:23.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:41:23.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:23.084 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:41:23.081+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:41:23.084 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:41:23.088 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:41:33.090 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:41:33.091 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:33.144 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:41:33.141+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:41:33.144 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:41:33.150 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:41:43.152 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:41:43.153 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:43.207 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:41:43.205+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:41:43.208 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:41:43.214 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:41:53.216 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:41:53.216 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule add -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:53.296 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:41:53.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd3/ns1 --image test1 2024-09-13T07:41:53.297 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2m' 2024-09-13T07:41:53.374 INFO:tasks.workunit.client.0.smithi005.stdout:every 1m, every 2m 2024-09-13T07:41:53.376 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd3/ns1 --image test1 2024-09-13T07:41:53.376 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:41:53.454 INFO:tasks.workunit.client.0.smithi005.stdout:every 1m, every 2m 2024-09-13T07:41:53.454 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule rm -p rbd3/ns1 --image test1 2m 2024-09-13T07:41:53.536 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule rm -p rbd3/ns1 --image test1 1m 2024-09-13T07:41:53.615 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd3/ns1 --image test1 2024-09-13T07:41:53.615 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep 'every 2m' 2024-09-13T07:41:53.615 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 2m' 2024-09-13T07:41:53.691 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:41:53.692 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror snapshot schedule ls -p rbd3/ns1 --image test1 2024-09-13T07:41:53.692 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep 'every 1m' 2024-09-13T07:41:53.693 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'every 1m' 2024-09-13T07:41:53.769 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:41:53.769 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap purge rbd3/ns1/test1 2024-09-13T07:41:53.840 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd rm rbd3/ns1/test1 2024-09-13T07:41:54.853 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:41:54.849+0000 7fe94bc75700 0 -- 172.21.15.5:0/2233940835 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x5563b1f43010 msgr2=0x5563b1f5c2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:41:54.873 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:41:54.869+0000 7fe94bc75700 0 -- 172.21.15.5:0/2233940835 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7fe93003abf0 msgr2=0x7fe93003cfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:41:54.936 INFO:tasks.workunit.client.0.smithi005.stderr: Removing image: 100% complete...done. 2024-09-13T07:41:54.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd3 rbd3 --yes-i-really-really-mean-it 2024-09-13T07:41:55.952 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd3' does not exist 2024-09-13T07:41:55.961 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_perf_image_iostat 2024-09-13T07:41:55.961 INFO:tasks.workunit.client.0.smithi005.stdout:testing perf image iostat... 2024-09-13T07:41:55.961 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing perf image iostat...' 2024-09-13T07:41:55.962 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:41:55.962 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:56.111 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:56.269 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:56.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:56.580 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:56.726 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:56.879 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:57.039 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:57.202 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:57.360 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:57.525 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:57.680 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:57.837 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:58.001 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:58.160 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:58.316 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:58.477 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:58.847 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:41:59.008 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd1 8 2024-09-13T07:41:59.615 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd1' already exists 2024-09-13T07:41:59.624 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd1 2024-09-13T07:42:01.783 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd1/ns 2024-09-13T07:42:01.835 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:42:02.892 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:42:02.900 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:42:05.811 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd2/ns 2024-09-13T07:42:05.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ IMAGE_SPECS=("test1" "rbd1/test2" "rbd1/ns/test3" "rbd2/test4" "rbd2/ns/test5") 2024-09-13T07:42:05.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:05.865 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' test1 2024-09-13T07:42:05.954 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:05.954 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' rbd1/test2 2024-09-13T07:42:06.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.031 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' rbd1/ns/test3 2024-09-13T07:42:06.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.322 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' rbd2/test4 2024-09-13T07:42:06.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.407 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' rbd2/ns/test5 2024-09-13T07:42:06.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS=() 2024-09-13T07:42:06.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:42:06.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.494 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false test1 2024-09-13T07:42:06.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:42:06.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false rbd1/test2 2024-09-13T07:42:06.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:42:06.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.495 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false rbd1/ns/test3 2024-09-13T07:42:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:42:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:42:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false rbd2/test4 2024-09-13T07:42:06.496 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:42:06.497 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false rbd2/ns/test5 2024-09-13T07:42:06.497 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json rbd1 2024-09-13T07:42:06.497 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:06.567 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:42:06.567 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:42:16.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test2 = test2 2024-09-13T07:42:16.586 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json rbd1/ns 2024-09-13T07:42:16.586 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:16.657 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:42:16.657 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:42:23.431 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:42:23.428+0000 7f3284771700 -1 reset not still connected to 0x5597ca5b4d00 2024-09-13T07:42:23.431 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-13T07:42:23.428+0000 7f3284771700 -1 reset not still connected to 0x5597cc31d040 2024-09-13T07:42:26.667 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test3 = test3 2024-09-13T07:42:26.667 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json --rbd-default-pool rbd1 /ns 2024-09-13T07:42:26.667 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:26.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test3 = test3 2024-09-13T07:42:26.749 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json --pool rbd2 2024-09-13T07:42:26.749 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:26.819 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:42:26.819 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:42:36.829 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test4 = test4 2024-09-13T07:42:36.831 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json --pool rbd2 --namespace ns 2024-09-13T07:42:36.832 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:36.902 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:42:36.902 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:42:46.916 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test5 = test5 2024-09-13T07:42:46.922 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json --rbd-default-pool rbd2 --namespace ns 2024-09-13T07:42:46.922 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:46.994 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test5 = test5 2024-09-13T07:42:46.995 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:42:46.995 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json 2024-09-13T07:42:47.058 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:42:47.059 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:42:57.331 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:42:57.319+0000 7ff392e7d700 -1 reset not still connected to 0x5583a0fc6f70 2024-09-13T07:43:02.075 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 'test1 test2 test3 test4 test5' = 'test1 test2 test3 test4 test5' 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 80274 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 80275 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 80276 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 80277 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:43:02.076 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 80278 2024-09-13T07:43:02.077 INFO:tasks.workunit.client.0.smithi005.stderr:+ wait 2024-09-13T07:43:02.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:43:02.125 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:02.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:02.426 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:02.593 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:02.755 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:02.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:03.091 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:03.256 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:03.425 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:03.591 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:03.768 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:03.934 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:04.096 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:04.260 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:06.790 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:06.949 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:07.117 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:07.290 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:07.457 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:43:08.006 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:43:08.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd1 rbd1 --yes-i-really-really-mean-it 2024-09-13T07:43:09.046 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd1' does not exist 2024-09-13T07:43:09.062 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_perf_image_iostat_recovery 2024-09-13T07:43:09.063 INFO:tasks.workunit.client.0.smithi005.stdout:testing recovery of perf handler after module's RADOS client is blocklisted... 2024-09-13T07:43:09.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing recovery of perf handler after module'\''s RADOS client is blocklisted...' 2024-09-13T07:43:09.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:43:09.063 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:09.240 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:09.401 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:09.566 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:09.738 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:09.905 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:10.060 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:10.219 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:10.399 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:10.569 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:10.739 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:10.917 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:11.080 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:11.252 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:11.423 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:11.585 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:11.762 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:11.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:43:12.099 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd3 8 2024-09-13T07:43:12.960 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd3' already exists 2024-09-13T07:43:12.969 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd3 2024-09-13T07:43:13.207 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:43:13.203+0000 7f405f48c700 0 --2- 172.21.15.5:0/3776544103 >> v2:172.21.15.5:3300/0 conn(0x56146226bc40 0x561462257550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:43:15.877 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd namespace create rbd3/ns 2024-09-13T07:43:15.932 INFO:tasks.workunit.client.0.smithi005.stderr:+ IMAGE_SPECS=("rbd3/test1" "rbd3/ns/test2") 2024-09-13T07:43:15.932 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:43:15.932 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' rbd3/test1 2024-09-13T07:43:16.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:43:16.014 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 10G --rbd-default-data-pool '' rbd3/ns/test2 2024-09-13T07:43:16.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS=() 2024-09-13T07:43:16.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:43:16.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:43:16.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ for spec in "${IMAGE_SPECS[@]}" 2024-09-13T07:43:16.101 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false rbd3/test1 2024-09-13T07:43:16.102 INFO:tasks.workunit.client.0.smithi005.stderr:+ BENCH_PIDS+=($!) 2024-09-13T07:43:16.102 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 10G --io-threads 1 --rbd-cache false rbd3/ns/test2 2024-09-13T07:43:16.103 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json rbd3 2024-09-13T07:43:16.104 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:43:16.163 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:43:16.163 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:43:16.591 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:43:16.587+0000 7ff392e7d700 -1 reset not still connected to 0x5583b2e64a90 2024-09-13T07:43:23.016 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:43:23.011+0000 7ff392e7d700 -1 reset not still connected to 0x5583b2e64a90 2024-09-13T07:43:24.997 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:43:24.991+0000 7ff392e7d700 -1 reset not still connected to 0x5583a3bec1a0 2024-09-13T07:43:24.997 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:43:24.991+0000 7ff392e7d700 -1 reset not still connected to 0x5583b2e64a90 2024-09-13T07:43:26.177 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test1 = test1 2024-09-13T07:43:26.178 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph mgr dump 2024-09-13T07:43:26.179 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq '.active_clients[]' 2024-09-13T07:43:26.179 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-13T07:43:26.182 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-13T07:43:26.868 INFO:tasks.workunit.client.0.smithi005.stderr:+ CLIENT_ADDR=172.21.15.5:0/2571982178 2024-09-13T07:43:26.869 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd blocklist add 172.21.15.5:0/2571982178 2024-09-13T07:43:28.798 INFO:tasks.workunit.client.0.smithi005.stderr:blocklisting 172.21.15.5:0/2571982178 until 2024-09-13T08:43:27.966903+0000 (3600 sec) 2024-09-13T07:43:28.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd perf image iostat --format json rbd3/ns 2024-09-13T07:43:28.813 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd perf image iostat --format json rbd3/ns 2024-09-13T07:43:28.870 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:43:28.870 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:43:33.868 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:43:33.863+0000 7fbaa41d9700 -1 librbd::api::Image: list_images_v2: error listing image in directory: (108) Cannot send after transport endpoint shutdown 2024-09-13T07:43:33.868 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:43:33.863+0000 7fbaa41d9700 -1 librbd::api::Image: list_images: error listing v2 images: (108) Cannot send after transport endpoint shutdown 2024-09-13T07:43:33.872 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:43:33.867+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:43:33.872 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: mgr command failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:43:33.876 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:43:33.876 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:43:43.879 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 24 2024-09-13T07:43:43.879 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:43:43.881 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json rbd3/ns 2024-09-13T07:43:43.882 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:43:43.934 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:43:43.930+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:43:43.935 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: mgr command failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:43:43.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '' = test2 2024-09-13T07:43:43.941 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:43:53.942 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:43:53.943 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:43:53.946 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json rbd3/ns 2024-09-13T07:43:54.001 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:43:53.998+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:43:54.001 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: mgr command failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2024-09-13T07:43:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '' = test2 2024-09-13T07:43:54.006 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:44:04.007 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:44:04.008 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd perf image iostat --format json rbd3/ns 2024-09-13T07:44:04.009 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r 'map(.image) | sort | join(" ")' 2024-09-13T07:44:04.064 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: waiting for initial image stats 2024-09-13T07:44:04.065 INFO:tasks.workunit.client.0.smithi005.stderr: 2024-09-13T07:44:10.350 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T07:44:10.346+0000 7ff392e7d700 -1 reset not still connected to 0x5583b2e64a90 2024-09-13T07:44:14.078 INFO:tasks.workunit.client.0.smithi005.stderr:+ test test2 = test2 2024-09-13T07:44:14.078 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:44:14.078 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:44:14.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 82222 2024-09-13T07:44:14.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ for pid in "${BENCH_PIDS[@]}" 2024-09-13T07:44:14.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ kill 82223 2024-09-13T07:44:14.079 INFO:tasks.workunit.client.0.smithi005.stderr:+ wait 2024-09-13T07:44:14.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:44:14.121 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:14.284 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:14.448 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:14.616 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:14.785 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:14.952 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:15.120 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:15.288 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:15.460 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:15.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:15.806 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:15.977 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:16.148 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:16.317 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:16.487 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:16.650 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:16.818 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:17.194 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:17.369 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd3 rbd3 --yes-i-really-really-mean-it 2024-09-13T07:44:18.030 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd3' does not exist 2024-09-13T07:44:18.044 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_mirror_pool_peer_bootstrap_create 2024-09-13T07:44:18.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing mirror pool peer bootstrap create...' 2024-09-13T07:44:18.045 INFO:tasks.workunit.client.0.smithi005.stdout:testing mirror pool peer bootstrap create... 2024-09-13T07:44:18.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:44:18.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:18.430 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:18.590 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:18.761 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:18.929 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:19.097 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:19.271 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:19.442 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:19.612 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:19.778 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:19.958 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:20.127 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:20.300 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:20.468 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:20.635 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:20.804 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:20.970 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:21.140 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:21.312 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd1 8 2024-09-13T07:44:21.971 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd1' already exists 2024-09-13T07:44:21.980 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd1 2024-09-13T07:44:24.892 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool enable rbd1 image 2024-09-13T07:44:24.947 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:44:26.009 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:44:26.017 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:44:28.919 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd mirror pool enable rbd2 pool 2024-09-13T07:44:28.982 INFO:tasks.workunit.client.0.smithi005.stderr:+ readarray -t MON_ADDRS 2024-09-13T07:44:28.983 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph mon dump 2024-09-13T07:44:28.983 INFO:tasks.workunit.client.0.smithi005.stderr:++ sed -n 's/^[0-9]: \(.*\) mon\.[a-z]$/\1/p' 2024-09-13T07:44:29.517 INFO:tasks.workunit.client.0.smithi005.stderr:dumped monmap epoch 1 2024-09-13T07:44:29.527 INFO:tasks.workunit.client.0.smithi005.stderr:+ BAD_MON_ADDR=1.2.3.4:6789 2024-09-13T07:44:29.527 INFO:tasks.workunit.client.0.smithi005.stderr:+ MON_HOST='[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],1.2.3.4:6789' 2024-09-13T07:44:29.529 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror pool peer bootstrap create --mon-host '[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],1.2.3.4:6789' rbd1 2024-09-13T07:44:29.529 INFO:tasks.workunit.client.0.smithi005.stderr:++ base64 -d 2024-09-13T07:44:29.591 INFO:tasks.workunit.client.0.smithi005.stderr:+ TOKEN='{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' 2024-09-13T07:44:29.591 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r .fsid 2024-09-13T07:44:29.636 INFO:tasks.workunit.client.0.smithi005.stderr:+ TOKEN_FSID=9dfd7139-cdcf-49aa-a58a-f69fdac1ee97 2024-09-13T07:44:29.636 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r .client_id 2024-09-13T07:44:29.681 INFO:tasks.workunit.client.0.smithi005.stderr:+ TOKEN_CLIENT_ID=rbd-mirror-peer 2024-09-13T07:44:29.681 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r .key 2024-09-13T07:44:29.724 INFO:tasks.workunit.client.0.smithi005.stderr:+ TOKEN_KEY=AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A== 2024-09-13T07:44:29.725 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r .mon_host 2024-09-13T07:44:29.764 INFO:tasks.workunit.client.0.smithi005.stderr:+ TOKEN_MON_HOST='[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]' 2024-09-13T07:44:29.765 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph fsid 2024-09-13T07:44:30.298 INFO:tasks.workunit.client.0.smithi005.stderr:+ test 9dfd7139-cdcf-49aa-a58a-f69fdac1ee97 = 9dfd7139-cdcf-49aa-a58a-f69fdac1ee97 2024-09-13T07:44:30.298 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph auth get-key client.rbd-mirror-peer 2024-09-13T07:44:30.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ test AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A== = AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A== 2024-09-13T07:44:30.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ for addr in "${MON_ADDRS[@]}" 2024-09-13T07:44:30.814 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep '[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]' 2024-09-13T07:44:30.816 INFO:tasks.workunit.client.0.smithi005.stdout:[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-13T07:44:30.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail fgrep 1.2.3.4:6789 2024-09-13T07:44:30.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ fgrep 1.2.3.4:6789 2024-09-13T07:44:30.819 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:44:30.820 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror pool peer bootstrap create --mon-host '[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],1.2.3.4:6789' rbd1 2024-09-13T07:44:30.820 INFO:tasks.workunit.client.0.smithi005.stderr:++ base64 -d 2024-09-13T07:44:30.869 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' = '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' 2024-09-13T07:44:30.870 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror pool peer bootstrap create rbd1 2024-09-13T07:44:30.870 INFO:tasks.workunit.client.0.smithi005.stderr:++ base64 -d 2024-09-13T07:44:30.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' = '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' 2024-09-13T07:44:30.924 INFO:tasks.workunit.client.0.smithi005.stderr:++ base64 -d 2024-09-13T07:44:30.924 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror pool peer bootstrap create --mon-host '[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],1.2.3.4:6789' rbd2 2024-09-13T07:44:30.974 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' = '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' 2024-09-13T07:44:30.975 INFO:tasks.workunit.client.0.smithi005.stderr:++ rbd mirror pool peer bootstrap create rbd2 2024-09-13T07:44:30.976 INFO:tasks.workunit.client.0.smithi005.stderr:++ base64 -d 2024-09-13T07:44:31.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' = '{"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","client_id":"rbd-mirror-peer","key":"AQBd7eNmtByhIhAAUMuQjMgFnuoaiv6H9tRJ+A==","mon_host":"[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]"}' 2024-09-13T07:44:31.027 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:44:32.029 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:44:32.038 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd1 rbd1 --yes-i-really-really-mean-it 2024-09-13T07:44:33.034 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd1' does not exist 2024-09-13T07:44:33.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_tasks_removed_pool 2024-09-13T07:44:33.043 INFO:tasks.workunit.client.0.smithi005.stdout:testing removing pool under running tasks... 2024-09-13T07:44:33.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing removing pool under running tasks...' 2024-09-13T07:44:33.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:44:33.043 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:33.199 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:33.367 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:33.541 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:33.710 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:33.878 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:34.045 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:34.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:34.387 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:34.555 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:34.730 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:34.891 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:35.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:35.235 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:35.406 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:35.576 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:35.746 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:35.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:36.098 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:44:36.624 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:44:36.633 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:44:39.564 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G foo 2024-09-13T07:44:39.666 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create foo@snap 2024-09-13T07:44:40.601 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T07:44:40.630 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect foo@snap 2024-09-13T07:44:40.775 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone foo@snap bar 2024-09-13T07:44:40.931 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G rbd2/dummy 2024-09-13T07:44:41.015 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern seq --io-size 1M --io-total 1G rbd2/dummy 2024-09-13T07:44:41.107 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential 2024-09-13T07:44:42.120 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:44:42.121 INFO:tasks.workunit.client.0.smithi005.stdout: 1 336 360.663 361 MiB/s 2024-09-13T07:44:43.110 INFO:tasks.workunit.client.0.smithi005.stdout: 2 656 341.47 341 MiB/s 2024-09-13T07:44:43.835 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:43.833+0000 7f7ed7892700 0 -- 172.21.15.5:0/82172079 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f7ebc03a780 msgr2=0x7f7ebc03cbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:44:44.117 INFO:tasks.workunit.client.0.smithi005.stdout: 3 992 339.172 339 MiB/s 2024-09-13T07:44:44.239 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 3 ops: 1024 ops/sec: 326.954 bytes/sec: 327 MiB/s 2024-09-13T07:44:44.251 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create rbd2/dummy@snap 2024-09-13T07:44:44.396 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete...2024-09-13T07:44:44.393+0000 7f7397a87700 0 -- 172.21.15.5:0/3367140411 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x7f737803a780 msgr2=0x7f737803cbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:44:44.421 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:44.417+0000 7f7397a87700 0 -- 172.21.15.5:0/3367140411 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55abc6f66a50 msgr2=0x7f737803e3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:44:44.573 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 100% complete...done. 2024-09-13T07:44:44.583 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect rbd2/dummy@snap 2024-09-13T07:44:44.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:44.684 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/dummy@snap rbd2/dummy1 2024-09-13T07:44:44.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:44.791 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/dummy@snap rbd2/dummy2 2024-09-13T07:44:44.908 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:44.908 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/dummy@snap rbd2/dummy3 2024-09-13T07:44:45.033 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:45.033 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/dummy@snap rbd2/dummy4 2024-09-13T07:44:45.154 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:45.155 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/dummy@snap rbd2/dummy5 2024-09-13T07:44:45.280 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd task list 2024-09-13T07:44:45.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:44:45.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:45.779 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/dummy1 2024-09-13T07:44:46.598 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 1, "id": "ee2034f8-3781-4fe8-a1fb-93e05eb60433", "message": "Flattening image rbd2/dummy1", "refs": {"action": "flatten", "pool_name": "rbd2", "pool_namespace": "", "image_name": "dummy1", "image_id": "32a4b9187e8"}, "in_progress": true, "progress": 0.08984375} 2024-09-13T07:44:46.631 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:46.631 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/dummy2 2024-09-13T07:44:47.901 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 2, "id": "c445b753-6a1b-451a-8d39-02c00bf53660", "message": "Flattening image rbd2/dummy2", "refs": {"action": "flatten", "pool_name": "rbd2", "pool_namespace": "", "image_name": "dummy2", "image_id": "32a719c7b0c0"}} 2024-09-13T07:44:47.929 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:47.929 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/dummy3 2024-09-13T07:44:48.912 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 3, "id": "8bc35994-76e2-4026-98c4-77b57afd386e", "message": "Flattening image rbd2/dummy3", "refs": {"action": "flatten", "pool_name": "rbd2", "pool_namespace": "", "image_name": "dummy3", "image_id": "32aa1dcb12bb"}} 2024-09-13T07:44:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:48.937 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/dummy4 2024-09-13T07:44:50.163 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 4, "id": "73f8273e-7494-40e3-aa8e-be5051484f8f", "message": "Flattening image rbd2/dummy4", "refs": {"action": "flatten", "pool_name": "rbd2", "pool_namespace": "", "image_name": "dummy4", "image_id": "32ad24f73465"}} 2024-09-13T07:44:50.189 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..5} 2024-09-13T07:44:50.189 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/dummy5 2024-09-13T07:44:51.345 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 5, "id": "2eebfb8a-fc48-4d8b-8ddc-751447f2861e", "message": "Flattening image rbd2/dummy5", "refs": {"action": "flatten", "pool_name": "rbd2", "pool_namespace": "", "image_name": "dummy5", "image_id": "32b09eb661eb"}} 2024-09-13T07:44:51.383 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool delete rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:44:52.835 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.833+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.836 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.833+0000 7fbaab1e7700 -1 librbd::object_map::InvalidateRequest: 0x55a552a9fa40 invalidating object map in-memory 2024-09-13T07:44:52.836 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.833+0000 7fbaab1e7700 -1 librbd::object_map::InvalidateRequest: 0x55a552a9fa40 invalidating object map on-disk 2024-09-13T07:44:52.836 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.833+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.836 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.833+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.851 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.849+0000 7fbaaa9e6700 -1 librbd::object_map::InvalidateRequest: 0x55a552a9fa40 should_complete: r=-2 2024-09-13T07:44:52.851 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.849+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to invalidate object map: (2) No such file or directory 2024-09-13T07:44:52.851 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.849+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.873 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.869+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.873 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.869+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.873 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.869+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.873 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.869+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.873 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.869+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.873 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.869+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.884 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.881+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.885 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.881+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.885 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.881+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.895 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.893+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.945 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.941+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.946 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.941+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.949 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.945+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.966 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.961+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.966 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.965+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.968 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.965+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.974 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.969+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.974 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.969+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:52.991 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.989+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.002 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:52.997+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.038 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:44:53.042 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.037+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.042 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.041+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.048 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.045+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.048 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.045+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.062 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.061+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.063 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd task list 2024-09-13T07:44:53.070 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.065+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.070 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.065+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.095 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.093+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.097 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.093+0000 7fbaaa9e6700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.108 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.105+0000 7fbaab1e7700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2024-09-13T07:44:53.188 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:44:53.185+0000 7fbaaa9e6700 -1 librbd::Journal: 0x55a551c36b00 handle_op_event_safe: failed to commit op event: (2) No such file or directory 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[ 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr: { 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr: "id": "73f8273e-7494-40e3-aa8e-be5051484f8f", 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr: "message": "Flattening image rbd2/dummy4", 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr: "refs": { 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr: "action": "flatten", 2024-09-13T07:44:53.577 INFO:tasks.workunit.client.0.smithi005.stderr: "image_id": "32ad24f73465", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "image_name": "dummy4", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "pool_name": "rbd2", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "pool_namespace": "" 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: }, 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "sequence": 4 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: }, 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: { 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "id": "2eebfb8a-fc48-4d8b-8ddc-751447f2861e", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "message": "Flattening image rbd2/dummy5", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "refs": { 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "action": "flatten", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "image_id": "32b09eb661eb", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "image_name": "dummy5", 2024-09-13T07:44:53.578 INFO:tasks.workunit.client.0.smithi005.stderr: "pool_name": "rbd2", 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr: "pool_namespace": "" 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr: }, 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr: "sequence": 5 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr: } 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr:]' '!=' '[]' 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info bar 2024-09-13T07:44:53.579 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: ' 2024-09-13T07:44:53.649 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd/foo@snap 2024-09-13T07:44:53.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail rbd snap unprotect foo@snap 2024-09-13T07:44:53.649 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect foo@snap 2024-09-13T07:44:53.753 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:53.749+0000 7ffb0cfbe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [3295d515251c] in pool 'rbd' 2024-09-13T07:44:53.753 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:53.749+0000 7ffb0cfbe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2024-09-13T07:44:53.753 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:53.749+0000 7ffb0cfbe700 -1 librbd::SnapshotUnprotectRequest: 0x7ffaf4016f40 should_complete_error: ret_val=-16 2024-09-13T07:44:53.760 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:53.757+0000 7ffb07fff700 -1 librbd::SnapshotUnprotectRequest: 0x7ffaf4016f40 should_complete_error: ret_val=-16 2024-09-13T07:44:53.773 INFO:tasks.workunit.client.0.smithi005.stderr:rbd: unprotecting snap failed: (16) Device or resource busy 2024-09-13T07:44:53.799 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:44:53.799 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten bar 2024-09-13T07:44:54.369 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 6, "id": "134ab56b-d8ec-4093-b2f3-342d3412ae7b", "message": "Flattening image rbd/bar", "refs": {"action": "flatten", "pool_name": "rbd", "pool_namespace": "", "image_name": "bar", "image_id": "3295d515251c"}, "retry_attempts": 1, "retry_time": "2024-09-13T07:45:24.327801"} 2024-09-13T07:44:54.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..12} 2024-09-13T07:44:54.385 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info bar 2024-09-13T07:44:54.386 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: ' 2024-09-13T07:44:54.443 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:44:54.444 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info bar 2024-09-13T07:44:54.444 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep 'parent: ' 2024-09-13T07:44:54.444 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: ' 2024-09-13T07:44:54.509 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:44:54.510 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect foo@snap 2024-09-13T07:44:54.862 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..12} 2024-09-13T07:44:54.863 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd task list 2024-09-13T07:44:55.183 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:44:55.181+0000 7fd36ed9d700 0 --2- 172.21.15.5:0/2497936276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd370126500 0x7fd37011b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T07:44:55.567 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:44:55.567 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:44:55.567 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd task list 2024-09-13T07:44:56.065 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:44:56.065 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:44:56.065 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:56.217 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:56.581 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:56.750 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:56.916 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:57.089 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:57.462 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:57.634 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:57.802 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:57.966 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:59.156 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:59.325 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:44:59.902 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:00.067 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:00.243 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:00.413 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:00.580 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:00.748 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:00.922 INFO:tasks.workunit.client.0.smithi005.stderr:+ test_tasks_recovery 2024-09-13T07:45:00.922 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo 'testing task handler recovery after module'\''s RADOS client is blocklisted...' 2024-09-13T07:45:00.922 INFO:tasks.workunit.client.0.smithi005.stdout:testing task handler recovery after module's RADOS client is blocklisted... 2024-09-13T07:45:00.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ remove_images 2024-09-13T07:45:00.923 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:01.090 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:01.256 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:01.426 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:01.587 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:01.760 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:01.933 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:02.109 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:02.289 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:02.463 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:02.640 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:02.805 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:02.975 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:03.147 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:03.312 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:03.482 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:03.653 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:03.816 INFO:tasks.workunit.client.0.smithi005.stderr:+ for img in $IMGS 2024-09-13T07:45:03.993 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool create rbd2 8 2024-09-13T07:45:04.955 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' already exists 2024-09-13T07:45:04.964 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd pool init rbd2 2024-09-13T07:45:07.821 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd create --image-format 2 --size 1G rbd2/img1 2024-09-13T07:45:07.907 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd bench --io-type write --io-pattern seq --io-size 1M --io-total 1G rbd2/img1 2024-09-13T07:45:07.999 INFO:tasks.workunit.client.0.smithi005.stdout:bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential 2024-09-13T07:45:09.020 INFO:tasks.workunit.client.0.smithi005.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-13T07:45:09.020 INFO:tasks.workunit.client.0.smithi005.stdout: 1 336 357.731 358 MiB/s 2024-09-13T07:45:09.287 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:45:09.285+0000 7fe1c2c6c700 0 -- 172.21.15.5:0/4078660919 >> [v2:172.21.15.5:6808/1121634097,v1:172.21.15.5:6809/1121634097] conn(0x55880dc68590 msgr2=0x55880dccce80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:45:09.840 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:45:09.837+0000 7fe1c1a0a700 0 -- 172.21.15.5:0/4078660919 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7fe1a403a8d0 msgr2=0x7fe1a403cd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T07:45:10.031 INFO:tasks.workunit.client.0.smithi005.stdout: 2 688 352.712 353 MiB/s 2024-09-13T07:45:11.003 INFO:tasks.workunit.client.0.smithi005.stdout:elapsed: 3 ops: 1024 ops/sec: 340.886 bytes/sec: 341 MiB/s 2024-09-13T07:45:11.016 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap create rbd2/img1@snap 2024-09-13T07:45:11.171 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 10% complete...2024-09-13T07:45:11.169+0000 7f776dab8700 0 -- 172.21.15.5:0/2998320119 >> [v2:172.21.15.5:6800/440298088,v1:172.21.15.5:6801/440298088] conn(0x7f7754016ce0 msgr2=0x7f7754012c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:45:11.201 INFO:tasks.workunit.client.0.smithi005.stderr:2024-09-13T07:45:11.197+0000 7f776d2b7700 0 -- 172.21.15.5:0/2998320119 >> [v2:172.21.15.5:6816/4236811925,v1:172.21.15.5:6817/4236811925] conn(0x7f7748005c00 msgr2=0x7f7748006010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T07:45:11.836 INFO:tasks.workunit.client.0.smithi005.stderr: Creating snap: 100% complete...done. 2024-09-13T07:45:11.852 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap protect rbd2/img1@snap 2024-09-13T07:45:11.968 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd clone rbd2/img1@snap rbd2/clone1 2024-09-13T07:45:12.085 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph mgr dump 2024-09-13T07:45:12.085 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq 'select(.name == "rbd_support")' 2024-09-13T07:45:12.086 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq '.active_clients[]' 2024-09-13T07:45:12.086 INFO:tasks.workunit.client.0.smithi005.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2024-09-13T07:45:12.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ CLIENT_ADDR=172.21.15.5:0/619231378 2024-09-13T07:45:12.611 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd blocklist add 172.21.15.5:0/619231378 2024-09-13T07:45:14.841 INFO:tasks.workunit.client.0.smithi005.stderr:blocklisting 172.21.15.5:0/619231378 until 2024-09-13T08:45:13.950460+0000 (3600 sec) 2024-09-13T07:45:14.853 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail ceph rbd task add flatten rbd2/clone1 2024-09-13T07:45:14.853 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/clone1 2024-09-13T07:45:15.204 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:15.201+0000 7fbaab1e7700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (108) Cannot send after transport endpoint shutdown 2024-09-13T07:45:15.204 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:15.201+0000 7fbaab1e7700 -1 librbd::ImageState: 0x55a552596780 failed to open image: (108) Cannot send after transport endpoint shutdown 2024-09-13T07:45:15.205 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:15.201+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable [errno 108] RBD connection was shutdown (error opening image b'clone1' at snapshot None) 2024-09-13T07:45:15.207 INFO:tasks.workunit.client.0.smithi005.stderr:Error EAGAIN: [errno 108] RBD connection was shutdown (error opening image b'clone1' at snapshot None) 2024-09-13T07:45:15.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:45:15.211 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:45:25.214 INFO:tasks.workunit.client.0.smithi005.stderr:++ seq 24 2024-09-13T07:45:25.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:45:25.215 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/clone1 2024-09-13T07:45:25.559 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:25.556+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:45:25.561 INFO:tasks.workunit.client.0.smithi005.stderr:Error EAGAIN: rbd_support module is not ready, try again 2024-09-13T07:45:25.565 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:45:35.567 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:45:35.567 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/clone1 2024-09-13T07:45:35.918 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:35.912+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:45:35.920 INFO:tasks.workunit.client.0.smithi005.stderr:Error EAGAIN: rbd_support module is not ready, try again 2024-09-13T07:45:35.924 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:45:45.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:45:45.926 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/clone1 2024-09-13T07:45:46.269 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:46.264+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:45:47.053 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:47.048+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:45:47.057 INFO:tasks.workunit.client.0.smithi005.stderr:Error EAGAIN: rbd_support module is not ready, try again 2024-09-13T07:45:47.061 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:45:57.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:45:57.064 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/clone1 2024-09-13T07:45:57.417 INFO:tasks.ceph.mgr.x.smithi005.stderr:2024-09-13T07:45:57.412+0000 7fbaa71df700 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2024-09-13T07:45:57.418 INFO:tasks.workunit.client.0.smithi005.stderr:Error EAGAIN: rbd_support module is not ready, try again 2024-09-13T07:45:57.422 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:46:07.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in `seq 24` 2024-09-13T07:46:07.424 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph rbd task add flatten rbd2/clone1 2024-09-13T07:46:08.226 INFO:tasks.workunit.client.0.smithi005.stdout:{"sequence": 1, "id": "9558614b-9609-4ba8-b1ef-7d83bd50920b", "message": "Flattening image rbd2/clone1", "refs": {"action": "flatten", "pool_name": "rbd2", "pool_namespace": "", "image_name": "clone1", "image_id": "33c9fdd0f9a"}, "in_progress": true, "progress": 0.08203125} 2024-09-13T07:46:08.267 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:46:08.267 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd task list 2024-09-13T07:46:09.112 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[ 2024-09-13T07:46:09.144 INFO:tasks.workunit.client.0.smithi005.stderr: { 2024-09-13T07:46:09.144 INFO:tasks.workunit.client.0.smithi005.stderr: "id": "9558614b-9609-4ba8-b1ef-7d83bd50920b", 2024-09-13T07:46:09.144 INFO:tasks.workunit.client.0.smithi005.stderr: "in_progress": true, 2024-09-13T07:46:09.144 INFO:tasks.workunit.client.0.smithi005.stderr: "message": "Flattening image rbd2/clone1", 2024-09-13T07:46:09.144 INFO:tasks.workunit.client.0.smithi005.stderr: "progress": 0.5703125, 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "refs": { 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "action": "flatten", 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "image_id": "33c9fdd0f9a", 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "image_name": "clone1", 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "pool_name": "rbd2", 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "pool_namespace": "" 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: }, 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: "sequence": 1 2024-09-13T07:46:09.145 INFO:tasks.workunit.client.0.smithi005.stderr: } 2024-09-13T07:46:09.146 INFO:tasks.workunit.client.0.smithi005.stderr:]' '!=' '[]' 2024-09-13T07:46:09.146 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..12} 2024-09-13T07:46:09.146 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info rbd2/clone1 2024-09-13T07:46:09.146 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: ' 2024-09-13T07:46:09.247 INFO:tasks.workunit.client.0.smithi005.stdout: parent: rbd2/img1@snap 2024-09-13T07:46:09.248 INFO:tasks.workunit.client.0.smithi005.stderr:+ sleep 10 2024-09-13T07:46:19.253 INFO:tasks.workunit.client.0.smithi005.stderr:+ for i in {1..12} 2024-09-13T07:46:19.253 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info rbd2/clone1 2024-09-13T07:46:19.253 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: ' 2024-09-13T07:46:19.324 INFO:tasks.workunit.client.0.smithi005.stderr:+ break 2024-09-13T07:46:19.325 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd info rbd2/clone1 2024-09-13T07:46:19.325 INFO:tasks.workunit.client.0.smithi005.stderr:+ expect_fail grep 'parent: ' 2024-09-13T07:46:19.325 INFO:tasks.workunit.client.0.smithi005.stderr:+ grep 'parent: ' 2024-09-13T07:46:19.392 INFO:tasks.workunit.client.0.smithi005.stderr:+ return 0 2024-09-13T07:46:19.392 INFO:tasks.workunit.client.0.smithi005.stderr:+ rbd snap unprotect rbd2/img1@snap 2024-09-13T07:46:19.513 INFO:tasks.workunit.client.0.smithi005.stderr:++ ceph rbd task list 2024-09-13T07:46:20.019 INFO:tasks.workunit.client.0.smithi005.stderr:+ test '[]' = '[]' 2024-09-13T07:46:20.019 INFO:tasks.workunit.client.0.smithi005.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2024-09-13T07:46:20.704 INFO:tasks.workunit.client.0.smithi005.stderr:pool 'rbd2' does not exist 2024-09-13T07:46:20.713 INFO:tasks.workunit.client.0.smithi005.stdout:OK 2024-09-13T07:46:20.715 INFO:tasks.workunit.client.0.smithi005.stderr:+ echo OK 2024-09-13T07:46:20.715 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T07:46:20.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T07:46:20.725 INFO:tasks.workunit:Stopping ['rbd/cli_generic.sh'] on client.0... 2024-09-13T07:46:20.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T07:46:21.041 DEBUG:teuthology.parallel:result is None 2024-09-13T07:46:21.042 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T07:46:21.058 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T07:46:21.058 DEBUG:teuthology.orchestra.run.smithi005:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T07:46:21.078 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T07:46:21.079 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T07:46:21.099 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T07:46:21.099 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:46:21.430 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:46:21.433 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:46:21.445 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-09-13T07:46:20.586869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443041358,"num_objects":1547,"num_object_clones":0,"num_object_copies":3094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":125715,"num_read_kb":4692373,"num_write":63654,"num_write_kb":15298218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65020,"ondisk_log_size":65020,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2742312,"kb_used_data":794256,"kb_used_omap":4,"kb_used_meta":1948027,"kb_avail":280373208,"statfs":{"total":289910292480,"available":287102164992,"internally_reserved":0,"allocated":813318144,"data_stored":12900911582,"data_compressed":57302974,"data_compressed_allocated":806125568,"data_compressed_original":12897689600,"omap_allocated":4782,"internal_metadata":1994779986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":682036094,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":-3807,"num_read_kb":735519,"num_write":-1668,"num_write_kb":-383171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.083163"},"pg_stats":[{"pgid":"2.7","version":"240'4260","reported_seq":10271,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806521+0000","last_change":"2024-09-13T07:25:49.892092+0000","last_active":"2024-09-13T07:45:15.806521+0000","last_peered":"2024-09-13T07:45:15.806521+0000","last_clean":"2024-09-13T07:45:15.806521+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:45:15.806521+0000","last_undegraded":"2024-09-13T07:45:15.806521+0000","last_fullsized":"2024-09-13T07:45:15.806521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:18.745059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15489,"num_read_kb":13863,"num_write":4046,"num_write_kb":6947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"240'3885","reported_seq":7871,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806421+0000","last_change":"2024-09-13T07:25:49.891916+0000","last_active":"2024-09-13T07:45:15.806421+0000","last_peered":"2024-09-13T07:45:15.806421+0000","last_clean":"2024-09-13T07:45:15.806421+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:45:15.806421+0000","last_undegraded":"2024-09-13T07:45:15.806421+0000","last_fullsized":"2024-09-13T07:45:15.806421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:09.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9132,"num_read_kb":27041,"num_write":2612,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"240'8013","reported_seq":12420,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.068797+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:45:15.068797+0000","last_peered":"2024-09-13T07:45:15.068797+0000","last_clean":"2024-09-13T07:45:15.068797+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:45:15.068797+0000","last_undegraded":"2024-09-13T07:45:15.068797+0000","last_fullsized":"2024-09-13T07:45:15.068797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:45.692840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30453,"num_read_kb":24411,"num_write":10477,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"240'6028","reported_seq":11442,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806116+0000","last_change":"2024-09-13T07:25:49.891182+0000","last_active":"2024-09-13T07:45:15.806116+0000","last_peered":"2024-09-13T07:45:15.806116+0000","last_clean":"2024-09-13T07:45:15.806116+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:45:15.806116+0000","last_undegraded":"2024-09-13T07:45:15.806116+0000","last_fullsized":"2024-09-13T07:45:15.806116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:57:22.870708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12268,"num_read_kb":10256,"num_write":4577,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"248'6308","reported_seq":7385,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.019575+0000","last_change":"2024-09-13T07:44:58.835608+0000","last_active":"2024-09-13T07:46:10.019575+0000","last_peered":"2024-09-13T07:46:10.019575+0000","last_clean":"2024-09-13T07:46:10.019575+0000","last_became_active":"2024-09-13T07:25:56.610999+0000","last_became_peered":"2024-09-13T07:25:56.610999+0000","last_unstale":"2024-09-13T07:46:10.019575+0000","last_undegraded":"2024-09-13T07:46:10.019575+0000","last_fullsized":"2024-09-13T07:46:10.019575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:22:20.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072277400000000003,"stat_sum":{"num_bytes":1690304512,"num_objects":404,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":857,"num_read_kb":1230055,"num_write":6463,"num_write_kb":3872623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"240'4230","reported_seq":11354,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:14.834730+0000","last_change":"2024-09-13T07:25:49.885232+0000","last_active":"2024-09-13T07:45:14.834730+0000","last_peered":"2024-09-13T07:45:14.834730+0000","last_clean":"2024-09-13T07:45:14.834730+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:45:14.834730+0000","last_undegraded":"2024-09-13T07:45:14.834730+0000","last_fullsized":"2024-09-13T07:45:14.834730+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:00.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15351,"num_read_kb":81517,"num_write":4129,"num_write_kb":39334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"248'5961","reported_seq":6966,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.959337+0000","last_change":"2024-09-13T07:44:58.830184+0000","last_active":"2024-09-13T07:46:09.959337+0000","last_peered":"2024-09-13T07:46:09.959337+0000","last_clean":"2024-09-13T07:46:09.959337+0000","last_became_active":"2024-09-13T07:25:56.611661+0000","last_became_peered":"2024-09-13T07:25:56.611661+0000","last_unstale":"2024-09-13T07:46:09.959337+0000","last_undegraded":"2024-09-13T07:46:09.959337+0000","last_fullsized":"2024-09-13T07:46:09.959337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:47:44.980987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010219020000000001,"stat_sum":{"num_bytes":1556086784,"num_objects":371,"num_object_clones":0,"num_object_copies":742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":770,"num_read_kb":1012264,"num_write":6150,"num_write_kb":3673038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"240'4472","reported_seq":9359,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:04.099276+0000","last_change":"2024-09-13T07:25:50.530745+0000","last_active":"2024-09-13T07:46:04.099276+0000","last_peered":"2024-09-13T07:46:04.099276+0000","last_clean":"2024-09-13T07:46:04.099276+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:46:04.099276+0000","last_undegraded":"2024-09-13T07:46:04.099276+0000","last_fullsized":"2024-09-13T07:46:04.099276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:25.418612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12405,"num_read_kb":27004,"num_write":4063,"num_write_kb":19318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"248'5934","reported_seq":6977,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.974913+0000","last_change":"2024-09-13T07:44:58.835404+0000","last_active":"2024-09-13T07:46:09.974913+0000","last_peered":"2024-09-13T07:46:09.974913+0000","last_clean":"2024-09-13T07:46:09.974913+0000","last_became_active":"2024-09-13T07:25:56.611595+0000","last_became_peered":"2024-09-13T07:25:56.611595+0000","last_unstale":"2024-09-13T07:46:09.974913+0000","last_undegraded":"2024-09-13T07:46:09.974913+0000","last_fullsized":"2024-09-13T07:46:09.974913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:53.213487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000575324,"stat_sum":{"num_bytes":1476395008,"num_objects":352,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":792,"num_read_kb":1043734,"num_write":6087,"num_write_kb":3719935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"240'3651","reported_seq":7591,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:16.281535+0000","last_change":"2024-09-13T07:25:50.530802+0000","last_active":"2024-09-13T07:45:16.281535+0000","last_peered":"2024-09-13T07:45:16.281535+0000","last_clean":"2024-09-13T07:45:16.281535+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:45:16.281535+0000","last_undegraded":"2024-09-13T07:45:16.281535+0000","last_fullsized":"2024-09-13T07:45:16.281535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:58:27.785140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8893,"num_read_kb":24443,"num_write":2579,"num_write_kb":13833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"248'6408","reported_seq":7825,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.012195+0000","last_change":"2024-09-13T07:44:58.830192+0000","last_active":"2024-09-13T07:46:10.012195+0000","last_peered":"2024-09-13T07:46:10.012195+0000","last_clean":"2024-09-13T07:46:10.012195+0000","last_became_active":"2024-09-13T07:25:56.612381+0000","last_became_peered":"2024-09-13T07:25:56.612381+0000","last_unstale":"2024-09-13T07:46:10.012195+0000","last_undegraded":"2024-09-13T07:46:10.012195+0000","last_fullsized":"2024-09-13T07:46:10.012195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:40:42.735066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091467999999999996,"stat_sum":{"num_bytes":1719664659,"num_objects":411,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":1208,"num_read_kb":1165451,"num_write":6619,"num_write_kb":3892857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"240'5773","reported_seq":12084,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.067874+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:45:15.067874+0000","last_peered":"2024-09-13T07:45:15.067874+0000","last_clean":"2024-09-13T07:45:15.067874+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:45:15.067874+0000","last_undegraded":"2024-09-13T07:45:15.067874+0000","last_fullsized":"2024-09-13T07:45:15.067874+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:42.260372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17991,"num_read_kb":32243,"num_write":5715,"num_write_kb":15977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":379,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.805293+0000","last_change":"2024-09-13T07:25:42.741998+0000","last_active":"2024-09-13T07:45:15.805293+0000","last_peered":"2024-09-13T07:45:15.805293+0000","last_clean":"2024-09-13T07:45:15.805293+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:45:15.805293+0000","last_undegraded":"2024-09-13T07:45:15.805293+0000","last_fullsized":"2024-09-13T07:45:15.805293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_clean_scrub_stamp":"2024-09-13T07:25:40.845594+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:10.726746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":6442450963,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":3627,"num_read_kb":4451504,"num_write":25319,"num_write_kb":15158453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805314560,"data_stored":12884901926,"data_compressed":57212928,"data_compressed_allocated":805306368,"data_compressed_original":12884901888,"omap_allocated":0,"internal_metadata":0},"log_size":24611,"ondisk_log_size":24611,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":121982,"num_read_kb":240778,"num_write":38198,"num_write_kb":137871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40312,"ondisk_log_size":40312,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673211,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840584,"kb_used_data":195840,"kb_used_omap":1,"kb_used_meta":644734,"kb_avail":93531256,"statfs":{"total":96636764160,"available":95776006144,"internally_reserved":0,"allocated":200540160,"data_stored":3171206263,"data_compressed":14086186,"data_compressed_allocated":198135808,"data_compressed_original":3170127872,"omap_allocated":1594,"internal_metadata":660208070},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45700000000000002,"15min":0.47899999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.621,"5min":2.569,"15min":2.569},"last":0.5},{"interface":"front","average":{"1min":0.748,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.46300000000000002,"5min":0.183,"15min":0.183},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Sep 13 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.55300000000000005},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.216},"max":{"1min":3.593,"5min":3.593,"15min":3.6389999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.626,"15min":0.58899999999999997},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":5.718,"5min":5.718,"15min":5.718},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1112696,"kb_used_data":395824,"kb_used_omap":1,"kb_used_meta":716862,"kb_avail":93259144,"statfs":{"total":96636764160,"available":95497363456,"internally_reserved":0,"allocated":405323776,"data_stored":6447971581,"data_compressed":28636374,"data_compressed_allocated":402939904,"data_compressed_original":6446907392,"omap_allocated":1594,"internal_metadata":734067142},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.47299999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.185},"max":{"1min":0.83399999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.191},"max":{"1min":0.879,"5min":0.879,"15min":2.153},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47899999999999998,"15min":0.55800000000000005},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.22500000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":2.4689999999999999},"last":0.496}]}]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789032,"kb_used_data":202592,"kb_used_omap":1,"kb_used_meta":586430,"kb_avail":93582808,"statfs":{"total":96636764160,"available":95828795392,"internally_reserved":0,"allocated":207454208,"data_stored":3281733738,"data_compressed":14580414,"data_compressed_allocated":205049856,"data_compressed_original":3280654336,"omap_allocated":1594,"internal_metadata":600504774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41299999999999998,"15min":0.47499999999999998},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.74199999999999999,"5min":0.92700000000000005,"15min":2.363},"last":0.442},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.67200000000000004,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.192,"15min":0.192},"max":{"1min":12.167,"5min":12.167,"15min":12.167},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.327},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.77200000000000002,"15min":0.69399999999999995},"min":{"1min":0.42399999999999999,"5min":0.26600000000000001,"15min":0.21099999999999999},"max":{"1min":9.9469999999999992,"5min":9.9469999999999992,"15min":9.9469999999999992},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197918720,"data_stored":3166699520,"data_compressed":14061120,"data_compressed_allocated":197918720,"data_compressed_original":3166699520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204738560,"data_stored":3275751443,"data_compressed":14545344,"data_compressed_allocated":204734464,"data_compressed_original":3275751424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402657280,"data_stored":6442450963,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:46:21.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:46:21.751 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:46:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:46:21.766 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-09-13T07:46:20.586869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443041358,"num_objects":1547,"num_object_clones":0,"num_object_copies":3094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":125715,"num_read_kb":4692373,"num_write":63654,"num_write_kb":15298218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65020,"ondisk_log_size":65020,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2742312,"kb_used_data":794256,"kb_used_omap":4,"kb_used_meta":1948027,"kb_avail":280373208,"statfs":{"total":289910292480,"available":287102164992,"internally_reserved":0,"allocated":813318144,"data_stored":12900911582,"data_compressed":57302974,"data_compressed_allocated":806125568,"data_compressed_original":12897689600,"omap_allocated":4782,"internal_metadata":1994779986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":682036094,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":-3807,"num_read_kb":735519,"num_write":-1668,"num_write_kb":-383171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.083163"},"pg_stats":[{"pgid":"2.7","version":"240'4260","reported_seq":10271,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806521+0000","last_change":"2024-09-13T07:25:49.892092+0000","last_active":"2024-09-13T07:45:15.806521+0000","last_peered":"2024-09-13T07:45:15.806521+0000","last_clean":"2024-09-13T07:45:15.806521+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:45:15.806521+0000","last_undegraded":"2024-09-13T07:45:15.806521+0000","last_fullsized":"2024-09-13T07:45:15.806521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:18.745059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15489,"num_read_kb":13863,"num_write":4046,"num_write_kb":6947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"240'3885","reported_seq":7871,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806421+0000","last_change":"2024-09-13T07:25:49.891916+0000","last_active":"2024-09-13T07:45:15.806421+0000","last_peered":"2024-09-13T07:45:15.806421+0000","last_clean":"2024-09-13T07:45:15.806421+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:45:15.806421+0000","last_undegraded":"2024-09-13T07:45:15.806421+0000","last_fullsized":"2024-09-13T07:45:15.806421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:09.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9132,"num_read_kb":27041,"num_write":2612,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"240'8013","reported_seq":12420,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.068797+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:45:15.068797+0000","last_peered":"2024-09-13T07:45:15.068797+0000","last_clean":"2024-09-13T07:45:15.068797+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:45:15.068797+0000","last_undegraded":"2024-09-13T07:45:15.068797+0000","last_fullsized":"2024-09-13T07:45:15.068797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:45.692840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30453,"num_read_kb":24411,"num_write":10477,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"240'6028","reported_seq":11442,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806116+0000","last_change":"2024-09-13T07:25:49.891182+0000","last_active":"2024-09-13T07:45:15.806116+0000","last_peered":"2024-09-13T07:45:15.806116+0000","last_clean":"2024-09-13T07:45:15.806116+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:45:15.806116+0000","last_undegraded":"2024-09-13T07:45:15.806116+0000","last_fullsized":"2024-09-13T07:45:15.806116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:57:22.870708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12268,"num_read_kb":10256,"num_write":4577,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"248'6308","reported_seq":7385,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.019575+0000","last_change":"2024-09-13T07:44:58.835608+0000","last_active":"2024-09-13T07:46:10.019575+0000","last_peered":"2024-09-13T07:46:10.019575+0000","last_clean":"2024-09-13T07:46:10.019575+0000","last_became_active":"2024-09-13T07:25:56.610999+0000","last_became_peered":"2024-09-13T07:25:56.610999+0000","last_unstale":"2024-09-13T07:46:10.019575+0000","last_undegraded":"2024-09-13T07:46:10.019575+0000","last_fullsized":"2024-09-13T07:46:10.019575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:22:20.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072277400000000003,"stat_sum":{"num_bytes":1690304512,"num_objects":404,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":857,"num_read_kb":1230055,"num_write":6463,"num_write_kb":3872623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"240'4230","reported_seq":11354,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:14.834730+0000","last_change":"2024-09-13T07:25:49.885232+0000","last_active":"2024-09-13T07:45:14.834730+0000","last_peered":"2024-09-13T07:45:14.834730+0000","last_clean":"2024-09-13T07:45:14.834730+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:45:14.834730+0000","last_undegraded":"2024-09-13T07:45:14.834730+0000","last_fullsized":"2024-09-13T07:45:14.834730+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:00.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15351,"num_read_kb":81517,"num_write":4129,"num_write_kb":39334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"248'5961","reported_seq":6966,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.959337+0000","last_change":"2024-09-13T07:44:58.830184+0000","last_active":"2024-09-13T07:46:09.959337+0000","last_peered":"2024-09-13T07:46:09.959337+0000","last_clean":"2024-09-13T07:46:09.959337+0000","last_became_active":"2024-09-13T07:25:56.611661+0000","last_became_peered":"2024-09-13T07:25:56.611661+0000","last_unstale":"2024-09-13T07:46:09.959337+0000","last_undegraded":"2024-09-13T07:46:09.959337+0000","last_fullsized":"2024-09-13T07:46:09.959337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:47:44.980987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010219020000000001,"stat_sum":{"num_bytes":1556086784,"num_objects":371,"num_object_clones":0,"num_object_copies":742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":770,"num_read_kb":1012264,"num_write":6150,"num_write_kb":3673038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"240'4472","reported_seq":9359,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:04.099276+0000","last_change":"2024-09-13T07:25:50.530745+0000","last_active":"2024-09-13T07:46:04.099276+0000","last_peered":"2024-09-13T07:46:04.099276+0000","last_clean":"2024-09-13T07:46:04.099276+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:46:04.099276+0000","last_undegraded":"2024-09-13T07:46:04.099276+0000","last_fullsized":"2024-09-13T07:46:04.099276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:25.418612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12405,"num_read_kb":27004,"num_write":4063,"num_write_kb":19318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"248'5934","reported_seq":6977,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.974913+0000","last_change":"2024-09-13T07:44:58.835404+0000","last_active":"2024-09-13T07:46:09.974913+0000","last_peered":"2024-09-13T07:46:09.974913+0000","last_clean":"2024-09-13T07:46:09.974913+0000","last_became_active":"2024-09-13T07:25:56.611595+0000","last_became_peered":"2024-09-13T07:25:56.611595+0000","last_unstale":"2024-09-13T07:46:09.974913+0000","last_undegraded":"2024-09-13T07:46:09.974913+0000","last_fullsized":"2024-09-13T07:46:09.974913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:53.213487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000575324,"stat_sum":{"num_bytes":1476395008,"num_objects":352,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":792,"num_read_kb":1043734,"num_write":6087,"num_write_kb":3719935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"240'3651","reported_seq":7591,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:16.281535+0000","last_change":"2024-09-13T07:25:50.530802+0000","last_active":"2024-09-13T07:45:16.281535+0000","last_peered":"2024-09-13T07:45:16.281535+0000","last_clean":"2024-09-13T07:45:16.281535+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:45:16.281535+0000","last_undegraded":"2024-09-13T07:45:16.281535+0000","last_fullsized":"2024-09-13T07:45:16.281535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:58:27.785140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8893,"num_read_kb":24443,"num_write":2579,"num_write_kb":13833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"248'6408","reported_seq":7825,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.012195+0000","last_change":"2024-09-13T07:44:58.830192+0000","last_active":"2024-09-13T07:46:10.012195+0000","last_peered":"2024-09-13T07:46:10.012195+0000","last_clean":"2024-09-13T07:46:10.012195+0000","last_became_active":"2024-09-13T07:25:56.612381+0000","last_became_peered":"2024-09-13T07:25:56.612381+0000","last_unstale":"2024-09-13T07:46:10.012195+0000","last_undegraded":"2024-09-13T07:46:10.012195+0000","last_fullsized":"2024-09-13T07:46:10.012195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:40:42.735066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091467999999999996,"stat_sum":{"num_bytes":1719664659,"num_objects":411,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":1208,"num_read_kb":1165451,"num_write":6619,"num_write_kb":3892857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"240'5773","reported_seq":12084,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.067874+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:45:15.067874+0000","last_peered":"2024-09-13T07:45:15.067874+0000","last_clean":"2024-09-13T07:45:15.067874+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:45:15.067874+0000","last_undegraded":"2024-09-13T07:45:15.067874+0000","last_fullsized":"2024-09-13T07:45:15.067874+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:42.260372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17991,"num_read_kb":32243,"num_write":5715,"num_write_kb":15977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":379,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.805293+0000","last_change":"2024-09-13T07:25:42.741998+0000","last_active":"2024-09-13T07:45:15.805293+0000","last_peered":"2024-09-13T07:45:15.805293+0000","last_clean":"2024-09-13T07:45:15.805293+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:45:15.805293+0000","last_undegraded":"2024-09-13T07:45:15.805293+0000","last_fullsized":"2024-09-13T07:45:15.805293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_clean_scrub_stamp":"2024-09-13T07:25:40.845594+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:10.726746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":6442450963,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":3627,"num_read_kb":4451504,"num_write":25319,"num_write_kb":15158453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805314560,"data_stored":12884901926,"data_compressed":57212928,"data_compressed_allocated":805306368,"data_compressed_original":12884901888,"omap_allocated":0,"internal_metadata":0},"log_size":24611,"ondisk_log_size":24611,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":121982,"num_read_kb":240778,"num_write":38198,"num_write_kb":137871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40312,"ondisk_log_size":40312,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673211,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840584,"kb_used_data":195840,"kb_used_omap":1,"kb_used_meta":644734,"kb_avail":93531256,"statfs":{"total":96636764160,"available":95776006144,"internally_reserved":0,"allocated":200540160,"data_stored":3171206263,"data_compressed":14086186,"data_compressed_allocated":198135808,"data_compressed_original":3170127872,"omap_allocated":1594,"internal_metadata":660208070},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45700000000000002,"15min":0.47899999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.621,"5min":2.569,"15min":2.569},"last":0.5},{"interface":"front","average":{"1min":0.748,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.46300000000000002,"5min":0.183,"15min":0.183},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Sep 13 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.55300000000000005},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.216},"max":{"1min":3.593,"5min":3.593,"15min":3.6389999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.626,"15min":0.58899999999999997},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":5.718,"5min":5.718,"15min":5.718},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1112696,"kb_used_data":395824,"kb_used_omap":1,"kb_used_meta":716862,"kb_avail":93259144,"statfs":{"total":96636764160,"available":95497363456,"internally_reserved":0,"allocated":405323776,"data_stored":6447971581,"data_compressed":28636374,"data_compressed_allocated":402939904,"data_compressed_original":6446907392,"omap_allocated":1594,"internal_metadata":734067142},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.47299999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.185},"max":{"1min":0.83399999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.191},"max":{"1min":0.879,"5min":0.879,"15min":2.153},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47899999999999998,"15min":0.55800000000000005},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.22500000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":2.4689999999999999},"last":0.496}]}]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789032,"kb_used_data":202592,"kb_used_omap":1,"kb_used_meta":586430,"kb_avail":93582808,"statfs":{"total":96636764160,"available":95828795392,"internally_reserved":0,"allocated":207454208,"data_stored":3281733738,"data_compressed":14580414,"data_compressed_allocated":205049856,"data_compressed_original":3280654336,"omap_allocated":1594,"internal_metadata":600504774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41299999999999998,"15min":0.47499999999999998},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.74199999999999999,"5min":0.92700000000000005,"15min":2.363},"last":0.442},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.67200000000000004,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.192,"15min":0.192},"max":{"1min":12.167,"5min":12.167,"15min":12.167},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.327},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.77200000000000002,"15min":0.69399999999999995},"min":{"1min":0.42399999999999999,"5min":0.26600000000000001,"15min":0.21099999999999999},"max":{"1min":9.9469999999999992,"5min":9.9469999999999992,"15min":9.9469999999999992},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197918720,"data_stored":3166699520,"data_compressed":14061120,"data_compressed_allocated":197918720,"data_compressed_original":3166699520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204738560,"data_stored":3275751443,"data_compressed":14545344,"data_compressed_allocated":204734464,"data_compressed_original":3275751424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402657280,"data_stored":6442450963,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:46:21.767 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T07:46:21.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:46:22.063 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:46:22.067 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:46:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-09-13T07:46:20.586869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443041358,"num_objects":1547,"num_object_clones":0,"num_object_copies":3094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":125715,"num_read_kb":4692373,"num_write":63654,"num_write_kb":15298218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65020,"ondisk_log_size":65020,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2742312,"kb_used_data":794256,"kb_used_omap":4,"kb_used_meta":1948027,"kb_avail":280373208,"statfs":{"total":289910292480,"available":287102164992,"internally_reserved":0,"allocated":813318144,"data_stored":12900911582,"data_compressed":57302974,"data_compressed_allocated":806125568,"data_compressed_original":12897689600,"omap_allocated":4782,"internal_metadata":1994779986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":682036094,"num_objects":146,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":-3807,"num_read_kb":735519,"num_write":-1668,"num_write_kb":-383171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.083163"},"pg_stats":[{"pgid":"2.7","version":"240'4260","reported_seq":10271,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806521+0000","last_change":"2024-09-13T07:25:49.892092+0000","last_active":"2024-09-13T07:45:15.806521+0000","last_peered":"2024-09-13T07:45:15.806521+0000","last_clean":"2024-09-13T07:45:15.806521+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:45:15.806521+0000","last_undegraded":"2024-09-13T07:45:15.806521+0000","last_fullsized":"2024-09-13T07:45:15.806521+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:18.745059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15489,"num_read_kb":13863,"num_write":4046,"num_write_kb":6947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"240'3885","reported_seq":7871,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806421+0000","last_change":"2024-09-13T07:25:49.891916+0000","last_active":"2024-09-13T07:45:15.806421+0000","last_peered":"2024-09-13T07:45:15.806421+0000","last_clean":"2024-09-13T07:45:15.806421+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:45:15.806421+0000","last_undegraded":"2024-09-13T07:45:15.806421+0000","last_fullsized":"2024-09-13T07:45:15.806421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:09.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9132,"num_read_kb":27041,"num_write":2612,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"240'8013","reported_seq":12420,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.068797+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:45:15.068797+0000","last_peered":"2024-09-13T07:45:15.068797+0000","last_clean":"2024-09-13T07:45:15.068797+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:45:15.068797+0000","last_undegraded":"2024-09-13T07:45:15.068797+0000","last_fullsized":"2024-09-13T07:45:15.068797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:45.692840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30453,"num_read_kb":24411,"num_write":10477,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"240'6028","reported_seq":11442,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806116+0000","last_change":"2024-09-13T07:25:49.891182+0000","last_active":"2024-09-13T07:45:15.806116+0000","last_peered":"2024-09-13T07:45:15.806116+0000","last_clean":"2024-09-13T07:45:15.806116+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:45:15.806116+0000","last_undegraded":"2024-09-13T07:45:15.806116+0000","last_fullsized":"2024-09-13T07:45:15.806116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:57:22.870708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12268,"num_read_kb":10256,"num_write":4577,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"248'6308","reported_seq":7385,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.019575+0000","last_change":"2024-09-13T07:44:58.835608+0000","last_active":"2024-09-13T07:46:10.019575+0000","last_peered":"2024-09-13T07:46:10.019575+0000","last_clean":"2024-09-13T07:46:10.019575+0000","last_became_active":"2024-09-13T07:25:56.610999+0000","last_became_peered":"2024-09-13T07:25:56.610999+0000","last_unstale":"2024-09-13T07:46:10.019575+0000","last_undegraded":"2024-09-13T07:46:10.019575+0000","last_fullsized":"2024-09-13T07:46:10.019575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:22:20.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072277400000000003,"stat_sum":{"num_bytes":1690304512,"num_objects":404,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":857,"num_read_kb":1230055,"num_write":6463,"num_write_kb":3872623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"240'4230","reported_seq":11354,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:14.834730+0000","last_change":"2024-09-13T07:25:49.885232+0000","last_active":"2024-09-13T07:45:14.834730+0000","last_peered":"2024-09-13T07:45:14.834730+0000","last_clean":"2024-09-13T07:45:14.834730+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:45:14.834730+0000","last_undegraded":"2024-09-13T07:45:14.834730+0000","last_fullsized":"2024-09-13T07:45:14.834730+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:00.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15351,"num_read_kb":81517,"num_write":4129,"num_write_kb":39334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"248'5961","reported_seq":6966,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.959337+0000","last_change":"2024-09-13T07:44:58.830184+0000","last_active":"2024-09-13T07:46:09.959337+0000","last_peered":"2024-09-13T07:46:09.959337+0000","last_clean":"2024-09-13T07:46:09.959337+0000","last_became_active":"2024-09-13T07:25:56.611661+0000","last_became_peered":"2024-09-13T07:25:56.611661+0000","last_unstale":"2024-09-13T07:46:09.959337+0000","last_undegraded":"2024-09-13T07:46:09.959337+0000","last_fullsized":"2024-09-13T07:46:09.959337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:47:44.980987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010219020000000001,"stat_sum":{"num_bytes":1556086784,"num_objects":371,"num_object_clones":0,"num_object_copies":742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":770,"num_read_kb":1012264,"num_write":6150,"num_write_kb":3673038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"240'4472","reported_seq":9359,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:04.099276+0000","last_change":"2024-09-13T07:25:50.530745+0000","last_active":"2024-09-13T07:46:04.099276+0000","last_peered":"2024-09-13T07:46:04.099276+0000","last_clean":"2024-09-13T07:46:04.099276+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:46:04.099276+0000","last_undegraded":"2024-09-13T07:46:04.099276+0000","last_fullsized":"2024-09-13T07:46:04.099276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:25.418612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12405,"num_read_kb":27004,"num_write":4063,"num_write_kb":19318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"248'5934","reported_seq":6977,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.974913+0000","last_change":"2024-09-13T07:44:58.835404+0000","last_active":"2024-09-13T07:46:09.974913+0000","last_peered":"2024-09-13T07:46:09.974913+0000","last_clean":"2024-09-13T07:46:09.974913+0000","last_became_active":"2024-09-13T07:25:56.611595+0000","last_became_peered":"2024-09-13T07:25:56.611595+0000","last_unstale":"2024-09-13T07:46:09.974913+0000","last_undegraded":"2024-09-13T07:46:09.974913+0000","last_fullsized":"2024-09-13T07:46:09.974913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:53.213487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000575324,"stat_sum":{"num_bytes":1476395008,"num_objects":352,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":792,"num_read_kb":1043734,"num_write":6087,"num_write_kb":3719935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"240'3651","reported_seq":7591,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:16.281535+0000","last_change":"2024-09-13T07:25:50.530802+0000","last_active":"2024-09-13T07:45:16.281535+0000","last_peered":"2024-09-13T07:45:16.281535+0000","last_clean":"2024-09-13T07:45:16.281535+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:45:16.281535+0000","last_undegraded":"2024-09-13T07:45:16.281535+0000","last_fullsized":"2024-09-13T07:45:16.281535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:58:27.785140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8893,"num_read_kb":24443,"num_write":2579,"num_write_kb":13833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"248'6408","reported_seq":7825,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.012195+0000","last_change":"2024-09-13T07:44:58.830192+0000","last_active":"2024-09-13T07:46:10.012195+0000","last_peered":"2024-09-13T07:46:10.012195+0000","last_clean":"2024-09-13T07:46:10.012195+0000","last_became_active":"2024-09-13T07:25:56.612381+0000","last_became_peered":"2024-09-13T07:25:56.612381+0000","last_unstale":"2024-09-13T07:46:10.012195+0000","last_undegraded":"2024-09-13T07:46:10.012195+0000","last_fullsized":"2024-09-13T07:46:10.012195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:40:42.735066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091467999999999996,"stat_sum":{"num_bytes":1719664659,"num_objects":411,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":1208,"num_read_kb":1165451,"num_write":6619,"num_write_kb":3892857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"240'5773","reported_seq":12084,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.067874+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:45:15.067874+0000","last_peered":"2024-09-13T07:45:15.067874+0000","last_clean":"2024-09-13T07:45:15.067874+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:45:15.067874+0000","last_undegraded":"2024-09-13T07:45:15.067874+0000","last_fullsized":"2024-09-13T07:45:15.067874+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:42.260372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17991,"num_read_kb":32243,"num_write":5715,"num_write_kb":15977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":379,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.805293+0000","last_change":"2024-09-13T07:25:42.741998+0000","last_active":"2024-09-13T07:45:15.805293+0000","last_peered":"2024-09-13T07:45:15.805293+0000","last_clean":"2024-09-13T07:45:15.805293+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:45:15.805293+0000","last_undegraded":"2024-09-13T07:45:15.805293+0000","last_fullsized":"2024-09-13T07:45:15.805293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_clean_scrub_stamp":"2024-09-13T07:25:40.845594+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:10.726746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":6442450963,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":3627,"num_read_kb":4451504,"num_write":25319,"num_write_kb":15158453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805314560,"data_stored":12884901926,"data_compressed":57212928,"data_compressed_allocated":805306368,"data_compressed_original":12884901888,"omap_allocated":0,"internal_metadata":0},"log_size":24611,"ondisk_log_size":24611,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":121982,"num_read_kb":240778,"num_write":38198,"num_write_kb":137871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40312,"ondisk_log_size":40312,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673211,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840584,"kb_used_data":195840,"kb_used_omap":1,"kb_used_meta":644734,"kb_avail":93531256,"statfs":{"total":96636764160,"available":95776006144,"internally_reserved":0,"allocated":200540160,"data_stored":3171206263,"data_compressed":14086186,"data_compressed_allocated":198135808,"data_compressed_original":3170127872,"omap_allocated":1594,"internal_metadata":660208070},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45700000000000002,"15min":0.47899999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.621,"5min":2.569,"15min":2.569},"last":0.5},{"interface":"front","average":{"1min":0.748,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.46300000000000002,"5min":0.183,"15min":0.183},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Sep 13 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.55300000000000005},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.216},"max":{"1min":3.593,"5min":3.593,"15min":3.6389999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.626,"15min":0.58899999999999997},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":5.718,"5min":5.718,"15min":5.718},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1112696,"kb_used_data":395824,"kb_used_omap":1,"kb_used_meta":716862,"kb_avail":93259144,"statfs":{"total":96636764160,"available":95497363456,"internally_reserved":0,"allocated":405323776,"data_stored":6447971581,"data_compressed":28636374,"data_compressed_allocated":402939904,"data_compressed_original":6446907392,"omap_allocated":1594,"internal_metadata":734067142},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.47299999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.185},"max":{"1min":0.83399999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.191},"max":{"1min":0.879,"5min":0.879,"15min":2.153},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47899999999999998,"15min":0.55800000000000005},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.22500000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":2.4689999999999999},"last":0.496}]}]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":789032,"kb_used_data":202592,"kb_used_omap":1,"kb_used_meta":586430,"kb_avail":93582808,"statfs":{"total":96636764160,"available":95828795392,"internally_reserved":0,"allocated":207454208,"data_stored":3281733738,"data_compressed":14580414,"data_compressed_allocated":205049856,"data_compressed_original":3280654336,"omap_allocated":1594,"internal_metadata":600504774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41299999999999998,"15min":0.47499999999999998},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.74199999999999999,"5min":0.92700000000000005,"15min":2.363},"last":0.442},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.67200000000000004,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.192,"15min":0.192},"max":{"1min":12.167,"5min":12.167,"15min":12.167},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.327},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.77200000000000002,"15min":0.69399999999999995},"min":{"1min":0.42399999999999999,"5min":0.26600000000000001,"15min":0.21099999999999999},"max":{"1min":9.9469999999999992,"5min":9.9469999999999992,"15min":9.9469999999999992},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197918720,"data_stored":3166699520,"data_compressed":14061120,"data_compressed_allocated":197918720,"data_compressed_original":3166699520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204738560,"data_stored":3275751443,"data_compressed":14545344,"data_compressed_allocated":204734464,"data_compressed_original":3275751424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402657280,"data_stored":6442450963,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:46:22.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T07:46:22.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:46:22.387 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":249,"fsid":"9dfd7139-cdcf-49aa-a58a-f69fdac1ee97","created":"2024-09-13T07:25:30.620079+0000","modified":"2024-09-13T07:46:20.582924+0000","last_up_change":"2024-09-13T07:25:39.843856+0000","last_in_change":"2024-09-13T07:25:31.512317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T07:25:40.241777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T07:25:46.036455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-13T07:25:55.501068+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":115,"snap_epoch":245,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f078194-fd93-489d-9396-3ca5def6fedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6809","nonce":1121634097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6811","nonce":1121634097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6815","nonce":1121634097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1121634097},{"type":"v1","addr":"172.21.15.5:6813","nonce":1121634097}]},"public_addr":"172.21.15.5:6809/1121634097","cluster_addr":"172.21.15.5:6811/1121634097","heartbeat_back_addr":"172.21.15.5:6815/1121634097","heartbeat_front_addr":"172.21.15.5:6813/1121634097","state":["exists","up"]},{"osd":1,"uuid":"4e401908-a2c9-4dfa-97dc-bc3ed56ca2fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6817","nonce":4236811925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6819","nonce":4236811925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6823","nonce":4236811925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":4236811925},{"type":"v1","addr":"172.21.15.5:6821","nonce":4236811925}]},"public_addr":"172.21.15.5:6817/4236811925","cluster_addr":"172.21.15.5:6819/4236811925","heartbeat_back_addr":"172.21.15.5:6823/4236811925","heartbeat_front_addr":"172.21.15.5:6821/4236811925","state":["exists","up"]},{"osd":2,"uuid":"9f34ffca-e475-4fc6-93f0-afb6dd3e7ef2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6801","nonce":440298088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6803","nonce":440298088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6807","nonce":440298088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":440298088},{"type":"v1","addr":"172.21.15.5:6805","nonce":440298088}]},"public_addr":"172.21.15.5:6801/440298088","cluster_addr":"172.21.15.5:6803/440298088","heartbeat_back_addr":"172.21.15.5:6807/440298088","heartbeat_front_addr":"172.21.15.5:6805/440298088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.660118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.468047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T07:25:36.373702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/619231378":"2024-09-13T08:45:12.926939+0000","172.21.15.5:0/2571982178":"2024-09-13T08:43:27.275726+0000","172.21.15.5:0/3795888129":"2024-09-13T08:36:15.519558+0000","172.21.15.5:0/3227060487":"2024-09-13T08:30:22.310202+0000","172.21.15.5:0/378276747":"2024-09-13T08:30:20.789676+0000","172.21.15.5:0/3975735010":"2024-09-13T08:30:23.871690+0000","172.21.15.5:0/345399310":"2024-09-13T08:29:45.840616+0000","172.21.15.5:0/3226133755":"2024-09-13T08:30:22.940378+0000","172.21.15.5:0/3035598857":"2024-09-13T08:29:44.933870+0000","172.21.15.5:0/2170969492":"2024-09-13T08:41:00.969579+0000","172.21.15.5:0/171968415":"2024-09-13T08:29:44.307469+0000","172.21.15.5:0/3577055810":"2024-09-13T08:29:42.957090+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":1}]},{"pool":16,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T07:46:23.402 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T07:46:23.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T07:46:23.563 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T07:46:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T07:46:23.564 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T07:46:23.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T07:46:23.897 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T07:46:23.907 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T07:46:23.907 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T07:46:24.056 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T07:46:24.057 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T07:46:24.057 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T07:46:24.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T07:46:24.372 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T07:46:24.385 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T07:46:24.386 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T07:46:24.538 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T07:46:24.538 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T07:46:24.538 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T07:46:24.555 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T07:46:24.868 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T07:46:24.883 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:46:25.214 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:46:25.217 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:46:25.228 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":869,"stamp":"2024-09-13T07:46:24.507664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443041358,"num_objects":1547,"num_object_clones":0,"num_object_copies":3094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":125715,"num_read_kb":4692373,"num_write":63654,"num_write_kb":15298218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65020,"ondisk_log_size":65020,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2742024,"kb_used_data":793664,"kb_used_omap":4,"kb_used_meta":1948027,"kb_avail":280373496,"statfs":{"total":289910292480,"available":287102459904,"internally_reserved":0,"allocated":812711936,"data_stored":12892120965,"data_compressed":57243976,"data_compressed_allocated":805560320,"data_compressed_original":12888903680,"omap_allocated":4782,"internal_metadata":1994779986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5829839,"num_objects":-20,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-20,"num_whiteouts":0,"num_read":-4847,"num_read_kb":-14633,"num_write":-2179,"num_write_kb":-1055093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002770"},"pg_stats":[{"pgid":"2.7","version":"240'4260","reported_seq":10272,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:19.286380+0000","last_change":"2024-09-13T07:25:49.892092+0000","last_active":"2024-09-13T07:45:19.286380+0000","last_peered":"2024-09-13T07:45:19.286380+0000","last_clean":"2024-09-13T07:45:19.286380+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:45:19.286380+0000","last_undegraded":"2024-09-13T07:45:19.286380+0000","last_fullsized":"2024-09-13T07:45:19.286380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:18.745059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15489,"num_read_kb":13863,"num_write":4046,"num_write_kb":6947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"240'3885","reported_seq":7872,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:19.286341+0000","last_change":"2024-09-13T07:25:49.891916+0000","last_active":"2024-09-13T07:45:19.286341+0000","last_peered":"2024-09-13T07:45:19.286341+0000","last_clean":"2024-09-13T07:45:19.286341+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:45:19.286341+0000","last_undegraded":"2024-09-13T07:45:19.286341+0000","last_fullsized":"2024-09-13T07:45:19.286341+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:09.490293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9132,"num_read_kb":27041,"num_write":2612,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"240'8013","reported_seq":12421,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806356+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:45:15.806356+0000","last_peered":"2024-09-13T07:45:15.806356+0000","last_clean":"2024-09-13T07:45:15.806356+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:45:15.806356+0000","last_undegraded":"2024-09-13T07:45:15.806356+0000","last_fullsized":"2024-09-13T07:45:15.806356+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:45.692840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30453,"num_read_kb":24411,"num_write":10477,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"240'6028","reported_seq":11443,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:04.100162+0000","last_change":"2024-09-13T07:25:49.891182+0000","last_active":"2024-09-13T07:46:04.100162+0000","last_peered":"2024-09-13T07:46:04.100162+0000","last_clean":"2024-09-13T07:46:04.100162+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:46:04.100162+0000","last_undegraded":"2024-09-13T07:46:04.100162+0000","last_fullsized":"2024-09-13T07:46:04.100162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:57:22.870708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12268,"num_read_kb":10256,"num_write":4577,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"248'6308","reported_seq":7386,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.020016+0000","last_change":"2024-09-13T07:44:58.835608+0000","last_active":"2024-09-13T07:46:10.020016+0000","last_peered":"2024-09-13T07:46:10.020016+0000","last_clean":"2024-09-13T07:46:10.020016+0000","last_became_active":"2024-09-13T07:25:56.610999+0000","last_became_peered":"2024-09-13T07:25:56.610999+0000","last_unstale":"2024-09-13T07:46:10.020016+0000","last_undegraded":"2024-09-13T07:46:10.020016+0000","last_fullsized":"2024-09-13T07:46:10.020016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T09:22:20.863941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072277400000000003,"stat_sum":{"num_bytes":1690304512,"num_objects":404,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":857,"num_read_kb":1230055,"num_write":6463,"num_write_kb":3872623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"240'4230","reported_seq":11355,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:16.234056+0000","last_change":"2024-09-13T07:25:49.885232+0000","last_active":"2024-09-13T07:45:16.234056+0000","last_peered":"2024-09-13T07:45:16.234056+0000","last_clean":"2024-09-13T07:45:16.234056+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:45:16.234056+0000","last_undegraded":"2024-09-13T07:45:16.234056+0000","last_fullsized":"2024-09-13T07:45:16.234056+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:46:00.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15351,"num_read_kb":81517,"num_write":4129,"num_write_kb":39334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"248'5961","reported_seq":6967,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:09.983395+0000","last_change":"2024-09-13T07:44:58.830184+0000","last_active":"2024-09-13T07:46:09.983395+0000","last_peered":"2024-09-13T07:46:09.983395+0000","last_clean":"2024-09-13T07:46:09.983395+0000","last_became_active":"2024-09-13T07:25:56.611661+0000","last_became_peered":"2024-09-13T07:25:56.611661+0000","last_unstale":"2024-09-13T07:46:09.983395+0000","last_undegraded":"2024-09-13T07:46:09.983395+0000","last_fullsized":"2024-09-13T07:46:09.983395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:47:44.980987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010219020000000001,"stat_sum":{"num_bytes":1556086784,"num_objects":371,"num_object_clones":0,"num_object_copies":742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":770,"num_read_kb":1012264,"num_write":6150,"num_write_kb":3673038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"240'4472","reported_seq":9360,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:04.106148+0000","last_change":"2024-09-13T07:25:50.530745+0000","last_active":"2024-09-13T07:46:04.106148+0000","last_peered":"2024-09-13T07:46:04.106148+0000","last_clean":"2024-09-13T07:46:04.106148+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:46:04.106148+0000","last_undegraded":"2024-09-13T07:46:04.106148+0000","last_fullsized":"2024-09-13T07:46:04.106148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:25.418612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12405,"num_read_kb":27004,"num_write":4063,"num_write_kb":19318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"248'5934","reported_seq":6978,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.001443+0000","last_change":"2024-09-13T07:44:58.835404+0000","last_active":"2024-09-13T07:46:10.001443+0000","last_peered":"2024-09-13T07:46:10.001443+0000","last_clean":"2024-09-13T07:46:10.001443+0000","last_became_active":"2024-09-13T07:25:56.611595+0000","last_became_peered":"2024-09-13T07:25:56.611595+0000","last_unstale":"2024-09-13T07:46:10.001443+0000","last_undegraded":"2024-09-13T07:46:10.001443+0000","last_fullsized":"2024-09-13T07:46:10.001443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:45:53.213487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000575324,"stat_sum":{"num_bytes":1476395008,"num_objects":352,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":792,"num_read_kb":1043734,"num_write":6087,"num_write_kb":3719935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"240'3651","reported_seq":7592,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:54.268755+0000","last_change":"2024-09-13T07:25:50.530802+0000","last_active":"2024-09-13T07:45:54.268755+0000","last_peered":"2024-09-13T07:45:54.268755+0000","last_clean":"2024-09-13T07:45:54.268755+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:45:54.268755+0000","last_undegraded":"2024-09-13T07:45:54.268755+0000","last_fullsized":"2024-09-13T07:45:54.268755+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T07:58:27.785140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8893,"num_read_kb":24443,"num_write":2579,"num_write_kb":13833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"248'6408","reported_seq":7826,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:46:10.015476+0000","last_change":"2024-09-13T07:44:58.830192+0000","last_active":"2024-09-13T07:46:10.015476+0000","last_peered":"2024-09-13T07:46:10.015476+0000","last_clean":"2024-09-13T07:46:10.015476+0000","last_became_active":"2024-09-13T07:25:56.612381+0000","last_became_peered":"2024-09-13T07:25:56.612381+0000","last_unstale":"2024-09-13T07:46:10.015476+0000","last_undegraded":"2024-09-13T07:46:10.015476+0000","last_fullsized":"2024-09-13T07:46:10.015476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:40:42.735066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091467999999999996,"stat_sum":{"num_bytes":1719664659,"num_objects":411,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":1208,"num_read_kb":1165451,"num_write":6619,"num_write_kb":3892857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"240'5773","reported_seq":12085,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:15.806002+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:45:15.806002+0000","last_peered":"2024-09-13T07:45:15.806002+0000","last_clean":"2024-09-13T07:45:15.806002+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:45:15.806002+0000","last_undegraded":"2024-09-13T07:45:15.806002+0000","last_fullsized":"2024-09-13T07:45:15.806002+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:42.260372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17991,"num_read_kb":32243,"num_write":5715,"num_write_kb":15977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":380,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T07:45:49.292310+0000","last_change":"2024-09-13T07:25:42.741998+0000","last_active":"2024-09-13T07:45:49.292310+0000","last_peered":"2024-09-13T07:45:49.292310+0000","last_clean":"2024-09-13T07:45:49.292310+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:45:49.292310+0000","last_undegraded":"2024-09-13T07:45:49.292310+0000","last_fullsized":"2024-09-13T07:45:49.292310+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:40.845594+0000","last_clean_scrub_stamp":"2024-09-13T07:25:40.845594+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:55:10.726746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":6442450963,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":3627,"num_read_kb":4451504,"num_write":25319,"num_write_kb":15158453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805314560,"data_stored":12884901926,"data_compressed":57212928,"data_compressed_allocated":805306368,"data_compressed_original":12884901888,"omap_allocated":0,"internal_metadata":0},"log_size":24611,"ondisk_log_size":24611,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":121982,"num_read_kb":240778,"num_write":38198,"num_write_kb":137871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40312,"ondisk_log_size":40312,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673212,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840584,"kb_used_data":195840,"kb_used_omap":1,"kb_used_meta":644734,"kb_avail":93531256,"statfs":{"total":96636764160,"available":95776006144,"internally_reserved":0,"allocated":200540160,"data_stored":3171206340,"data_compressed":14086186,"data_compressed_allocated":198135808,"data_compressed_original":3170127872,"omap_allocated":1594,"internal_metadata":660208070},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45700000000000002,"15min":0.47899999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.621,"5min":2.569,"15min":2.569},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.46300000000000002,"5min":0.183,"15min":0.183},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.40300000000000002}]},{"osd":2,"last update":"Fri Sep 13 07:46:16 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.54000000000000004,"15min":0.56000000000000005},"min":{"1min":0.51200000000000001,"5min":0.222,"15min":0.216},"max":{"1min":1.1919999999999999,"5min":3.593,"15min":3.6389999999999998},"last":0.437},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.52500000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.97999999999999998,"5min":5.718,"15min":5.718},"last":0.51200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705917,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1112692,"kb_used_data":395520,"kb_used_omap":1,"kb_used_meta":716862,"kb_avail":93259148,"statfs":{"total":96636764160,"available":95497367552,"internally_reserved":0,"allocated":405012480,"data_stored":6443511875,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":1594,"internal_metadata":734067142},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.47299999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.185},"max":{"1min":0.83399999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.191},"max":{"1min":0.879,"5min":0.879,"15min":2.153},"last":0.94499999999999995}]},{"osd":1,"last update":"Fri Sep 13 07:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47899999999999998,"15min":0.55800000000000005},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.22500000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":2.4689999999999999},"last":0.85699999999999998}]}]},{"osd":1,"up_from":9,"seq":38654705915,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788748,"kb_used_data":202304,"kb_used_omap":1,"kb_used_meta":586430,"kb_avail":93583092,"statfs":{"total":96636764160,"available":95829086208,"internally_reserved":0,"allocated":207159296,"data_stored":3277402750,"data_compressed":14551326,"data_compressed_allocated":204771328,"data_compressed_original":3276324864,"omap_allocated":1594,"internal_metadata":600504774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41299999999999998,"15min":0.47499999999999998},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.74199999999999999,"5min":0.92700000000000005,"15min":2.363},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.67200000000000004,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.192,"15min":0.192},"max":{"1min":12.167,"5min":12.167,"15min":12.167},"last":0.497}]},{"osd":2,"last update":"Fri Sep 13 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.327},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.77200000000000002,"15min":0.69399999999999995},"min":{"1min":0.42399999999999999,"5min":0.26600000000000001,"15min":0.21099999999999999},"max":{"1min":9.9469999999999992,"5min":9.9469999999999992,"15min":9.9469999999999992},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197918720,"data_stored":3166699520,"data_compressed":14061120,"data_compressed_allocated":197918720,"data_compressed_original":3166699520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204738560,"data_stored":3275751443,"data_compressed":14545344,"data_compressed_allocated":204734464,"data_compressed_original":3275751424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402657280,"data_stored":6442450963,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:46:25.232 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.232 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-13T07:25:55.596315+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-13T07:25:55.596315+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-13T07:25:55.596315+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.233 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.234 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-13T07:25:55.596315+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.234 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.234 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T07:25:40.845594+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:25.234 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T07:46:45.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:46:45.541 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:46:45.544 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:46:45.560 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":879,"stamp":"2024-09-13T07:46:44.512753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443041358,"num_objects":1547,"num_object_clones":0,"num_object_copies":3094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":125715,"num_read_kb":4692373,"num_write":63654,"num_write_kb":15298218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65670,"ondisk_log_size":65670,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2767764,"kb_used_data":793468,"kb_used_omap":4,"kb_used_meta":1974267,"kb_avail":280347756,"statfs":{"total":289910292480,"available":287076102144,"internally_reserved":0,"allocated":812511232,"data_stored":12889265452,"data_compressed":57224892,"data_compressed_allocated":805380096,"data_compressed_original":12886048768,"omap_allocated":4782,"internal_metadata":2021649746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"2.7","version":"240'4260","reported_seq":10284,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:42.255162+0000","last_change":"2024-09-13T07:46:42.255430+0000","last_active":"2024-09-13T07:46:42.255162+0000","last_peered":"2024-09-13T07:46:42.255162+0000","last_clean":"2024-09-13T07:46:42.255162+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:46:42.255162+0000","last_undegraded":"2024-09-13T07:46:42.255162+0000","last_fullsized":"2024-09-13T07:46:42.255162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'4260","last_scrub_stamp":"2024-09-13T07:46:42.255106+0000","last_deep_scrub":"240'4260","last_deep_scrub_stamp":"2024-09-13T07:46:42.255106+0000","last_clean_scrub_stamp":"2024-09-13T07:46:42.255106+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:03:58.846624+0000","scrub_duration":0.0069620580000000001,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15489,"num_read_kb":13863,"num_write":4046,"num_write_kb":6947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"240'3885","reported_seq":7884,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:41.218763+0000","last_change":"2024-09-13T07:46:41.219031+0000","last_active":"2024-09-13T07:46:41.218763+0000","last_peered":"2024-09-13T07:46:41.218763+0000","last_clean":"2024-09-13T07:46:41.218763+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:46:41.218763+0000","last_undegraded":"2024-09-13T07:46:41.218763+0000","last_fullsized":"2024-09-13T07:46:41.218763+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'3885","last_scrub_stamp":"2024-09-13T07:46:41.218712+0000","last_deep_scrub":"240'3885","last_deep_scrub_stamp":"2024-09-13T07:46:41.218712+0000","last_clean_scrub_stamp":"2024-09-13T07:46:41.218712+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:50.774341+0000","scrub_duration":0.0065878200000000003,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9132,"num_read_kb":27041,"num_write":2612,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"240'8013","reported_seq":12423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:24.368739+0000","last_change":"2024-09-13T07:25:49.891594+0000","last_active":"2024-09-13T07:46:24.368739+0000","last_peered":"2024-09-13T07:46:24.368739+0000","last_clean":"2024-09-13T07:46:24.368739+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:46:24.368739+0000","last_undegraded":"2024-09-13T07:46:24.368739+0000","last_fullsized":"2024-09-13T07:46:24.368739+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30453,"num_read_kb":24411,"num_write":10477,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"249'6030","reported_seq":11458,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:27.112265+0000","last_change":"2024-09-13T07:46:27.112265+0000","last_active":"2024-09-13T07:46:27.112265+0000","last_peered":"2024-09-13T07:46:27.112265+0000","last_clean":"2024-09-13T07:46:27.112265+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:46:27.112265+0000","last_undegraded":"2024-09-13T07:46:27.112265+0000","last_fullsized":"2024-09-13T07:46:27.112265+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6030","last_scrub_stamp":"2024-09-13T07:46:27.112008+0000","last_deep_scrub":"249'6030","last_deep_scrub_stamp":"2024-09-13T07:46:27.112008+0000","last_clean_scrub_stamp":"2024-09-13T07:46:27.112008+0000","objects_scrubbed":2,"log_size":6030,"log_dups_size":0,"ondisk_log_size":6030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:41.319842+0000","scrub_duration":0.015654925,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12268,"num_read_kb":10256,"num_write":4577,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"249'6656","reported_seq":7779,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:32.991095+0000","last_change":"2024-09-13T07:46:32.991208+0000","last_active":"2024-09-13T07:46:32.991095+0000","last_peered":"2024-09-13T07:46:32.991095+0000","last_clean":"2024-09-13T07:46:32.991095+0000","last_became_active":"2024-09-13T07:25:56.610999+0000","last_became_peered":"2024-09-13T07:25:56.610999+0000","last_unstale":"2024-09-13T07:46:32.991095+0000","last_undegraded":"2024-09-13T07:46:32.991095+0000","last_fullsized":"2024-09-13T07:46:32.991095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6656","last_scrub_stamp":"2024-09-13T07:46:32.991074+0000","last_deep_scrub":"249'6656","last_deep_scrub_stamp":"2024-09-13T07:46:32.991074+0000","last_clean_scrub_stamp":"2024-09-13T07:46:32.991074+0000","objects_scrubbed":404,"log_size":6656,"log_dups_size":0,"ondisk_log_size":6656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:03:40.531248+0000","scrub_duration":2.8505866659999999,"objects_trimmed":0,"snaptrim_duration":0.00072277400000000003,"stat_sum":{"num_bytes":1690304512,"num_objects":404,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":857,"num_read_kb":1230055,"num_write":6463,"num_write_kb":3872623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"249'4232","reported_seq":11370,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:24.142391+0000","last_change":"2024-09-13T07:46:24.142391+0000","last_active":"2024-09-13T07:46:24.142391+0000","last_peered":"2024-09-13T07:46:24.142391+0000","last_clean":"2024-09-13T07:46:24.142391+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:46:24.142391+0000","last_undegraded":"2024-09-13T07:46:24.142391+0000","last_fullsized":"2024-09-13T07:46:24.142391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'4232","last_scrub_stamp":"2024-09-13T07:46:24.141846+0000","last_deep_scrub":"249'4232","last_deep_scrub_stamp":"2024-09-13T07:46:24.141846+0000","last_clean_scrub_stamp":"2024-09-13T07:46:24.141846+0000","objects_scrubbed":2,"log_size":4232,"log_dups_size":0,"ondisk_log_size":4232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:55:34.383253+0000","scrub_duration":0.017522368,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15351,"num_read_kb":81517,"num_write":4129,"num_write_kb":39334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"248'5961","reported_seq":6968,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:22.460456+0000","last_change":"2024-09-13T07:44:58.830184+0000","last_active":"2024-09-13T07:46:22.460456+0000","last_peered":"2024-09-13T07:46:22.460456+0000","last_clean":"2024-09-13T07:46:22.460456+0000","last_became_active":"2024-09-13T07:25:56.611661+0000","last_became_peered":"2024-09-13T07:25:56.611661+0000","last_unstale":"2024-09-13T07:46:22.460456+0000","last_undegraded":"2024-09-13T07:46:22.460456+0000","last_fullsized":"2024-09-13T07:46:22.460456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010219020000000001,"stat_sum":{"num_bytes":1556086784,"num_objects":371,"num_object_clones":0,"num_object_copies":742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":770,"num_read_kb":1012264,"num_write":6150,"num_write_kb":3673038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"249'4474","reported_seq":9375,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:32.232424+0000","last_change":"2024-09-13T07:46:32.232424+0000","last_active":"2024-09-13T07:46:32.232424+0000","last_peered":"2024-09-13T07:46:32.232424+0000","last_clean":"2024-09-13T07:46:32.232424+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:46:32.232424+0000","last_undegraded":"2024-09-13T07:46:32.232424+0000","last_fullsized":"2024-09-13T07:46:32.232424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'4474","last_scrub_stamp":"2024-09-13T07:46:32.232307+0000","last_deep_scrub":"249'4474","last_deep_scrub_stamp":"2024-09-13T07:46:32.232307+0000","last_clean_scrub_stamp":"2024-09-13T07:46:32.232307+0000","objects_scrubbed":2,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:14:33.145875+0000","scrub_duration":0.012268091,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12405,"num_read_kb":27004,"num_write":4063,"num_write_kb":19318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"249'6228","reported_seq":7313,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:27.525411+0000","last_change":"2024-09-13T07:46:27.525591+0000","last_active":"2024-09-13T07:46:27.525411+0000","last_peered":"2024-09-13T07:46:27.525411+0000","last_clean":"2024-09-13T07:46:27.525411+0000","last_became_active":"2024-09-13T07:25:56.611595+0000","last_became_peered":"2024-09-13T07:25:56.611595+0000","last_unstale":"2024-09-13T07:46:27.525411+0000","last_undegraded":"2024-09-13T07:46:27.525411+0000","last_fullsized":"2024-09-13T07:46:27.525411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6228","last_scrub_stamp":"2024-09-13T07:46:27.525376+0000","last_deep_scrub":"249'6228","last_deep_scrub_stamp":"2024-09-13T07:46:27.525376+0000","last_clean_scrub_stamp":"2024-09-13T07:46:27.525376+0000","objects_scrubbed":352,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:04.530529+0000","scrub_duration":2.4237470619999999,"objects_trimmed":0,"snaptrim_duration":0.000575324,"stat_sum":{"num_bytes":1476395008,"num_objects":352,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":792,"num_read_kb":1043734,"num_write":6087,"num_write_kb":3719935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"240'3651","reported_seq":7604,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:29.185372+0000","last_change":"2024-09-13T07:46:29.185761+0000","last_active":"2024-09-13T07:46:29.185372+0000","last_peered":"2024-09-13T07:46:29.185372+0000","last_clean":"2024-09-13T07:46:29.185372+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:46:29.185372+0000","last_undegraded":"2024-09-13T07:46:29.185372+0000","last_fullsized":"2024-09-13T07:46:29.185372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'3651","last_scrub_stamp":"2024-09-13T07:46:29.185264+0000","last_deep_scrub":"240'3651","last_deep_scrub_stamp":"2024-09-13T07:46:29.185264+0000","last_clean_scrub_stamp":"2024-09-13T07:46:29.185264+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:20.455557+0000","scrub_duration":0.0074364490000000004,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8893,"num_read_kb":24443,"num_write":2579,"num_write_kb":13833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"248'6408","reported_seq":7827,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:22.460693+0000","last_change":"2024-09-13T07:44:58.830192+0000","last_active":"2024-09-13T07:46:22.460693+0000","last_peered":"2024-09-13T07:46:22.460693+0000","last_clean":"2024-09-13T07:46:22.460693+0000","last_became_active":"2024-09-13T07:25:56.612381+0000","last_became_peered":"2024-09-13T07:25:56.612381+0000","last_unstale":"2024-09-13T07:46:22.460693+0000","last_undegraded":"2024-09-13T07:46:22.460693+0000","last_fullsized":"2024-09-13T07:46:22.460693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:55.596315+0000","last_clean_scrub_stamp":"2024-09-13T07:25:55.596315+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091467999999999996,"stat_sum":{"num_bytes":1719664659,"num_objects":411,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":1208,"num_read_kb":1165451,"num_write":6619,"num_write_kb":3892857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"240'5773","reported_seq":12087,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:24.368717+0000","last_change":"2024-09-13T07:25:49.891222+0000","last_active":"2024-09-13T07:46:24.368717+0000","last_peered":"2024-09-13T07:46:24.368717+0000","last_clean":"2024-09-13T07:46:24.368717+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:46:24.368717+0000","last_undegraded":"2024-09-13T07:46:24.368717+0000","last_fullsized":"2024-09-13T07:46:24.368717+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T07:25:46.862483+0000","last_clean_scrub_stamp":"2024-09-13T07:25:46.862483+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17991,"num_read_kb":32243,"num_write":5715,"num_write_kb":15977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"249'99","reported_seq":394,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:26.071981+0000","last_change":"2024-09-13T07:46:26.072305+0000","last_active":"2024-09-13T07:46:26.071981+0000","last_peered":"2024-09-13T07:46:26.071981+0000","last_clean":"2024-09-13T07:46:26.071981+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:46:26.071981+0000","last_undegraded":"2024-09-13T07:46:26.071981+0000","last_fullsized":"2024-09-13T07:46:26.071981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'99","last_scrub_stamp":"2024-09-13T07:46:26.071916+0000","last_deep_scrub":"249'99","last_deep_scrub_stamp":"2024-09-13T07:46:26.071916+0000","last_clean_scrub_stamp":"2024-09-13T07:46:26.071916+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:13:31.823771+0000","scrub_duration":0.0066743669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":6442450963,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":3627,"num_read_kb":4451504,"num_write":25319,"num_write_kb":15158453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805314560,"data_stored":12884901926,"data_compressed":57212928,"data_compressed_allocated":805306368,"data_compressed_original":12884901888,"omap_allocated":0,"internal_metadata":0},"log_size":25253,"ondisk_log_size":25253,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":121982,"num_read_kb":240778,"num_write":38198,"num_write_kb":137871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40318,"ondisk_log_size":40318,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862532,"kb_used_data":195644,"kb_used_omap":1,"kb_used_meta":666878,"kb_avail":93509308,"statfs":{"total":96636764160,"available":95753531392,"internally_reserved":0,"allocated":200339456,"data_stored":3168350827,"data_compressed":14067102,"data_compressed_allocated":197955584,"data_compressed_original":3167272960,"omap_allocated":1594,"internal_metadata":682883526},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45700000000000002,"15min":0.47899999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.621,"5min":2.569,"15min":2.569},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.46300000000000002,"5min":0.183,"15min":0.183},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Sep 13 07:46:16 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.54000000000000004,"15min":0.56000000000000005},"min":{"1min":0.51200000000000001,"5min":0.222,"15min":0.216},"max":{"1min":1.1919999999999999,"5min":3.593,"15min":3.6389999999999998},"last":0.499},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.52500000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.97999999999999998,"5min":5.718,"15min":5.718},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705921,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1116488,"kb_used_data":395520,"kb_used_omap":1,"kb_used_meta":720958,"kb_avail":93255352,"statfs":{"total":96636764160,"available":95493480448,"internally_reserved":0,"allocated":405012480,"data_stored":6443511875,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":1594,"internal_metadata":738261446},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.438,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.186,"15min":0.185},"max":{"1min":0.67300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.51400000000000001,"15min":0.55900000000000005},"min":{"1min":0.47299999999999998,"5min":0.217,"15min":0.191},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":2.153},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Sep 13 07:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.29399999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68300000000000005,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.48199999999999998,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.23000000000000001,"15min":0.22500000000000001},"max":{"1min":0.85699999999999998,"5min":1.1819999999999999,"15min":2.4689999999999999},"last":0.64300000000000002}]}]},{"osd":1,"up_from":9,"seq":38654705919,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788744,"kb_used_data":202304,"kb_used_omap":1,"kb_used_meta":586430,"kb_avail":93583096,"statfs":{"total":96636764160,"available":95829090304,"internally_reserved":0,"allocated":207159296,"data_stored":3277402750,"data_compressed":14551326,"data_compressed_allocated":204771328,"data_compressed_original":3276324864,"omap_allocated":1594,"internal_metadata":600504774},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.41599999999999998,"15min":0.47699999999999998},"min":{"1min":0.442,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.69899999999999995,"5min":0.92700000000000005,"15min":2.363},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.67200000000000004,"15min":0.58799999999999997},"min":{"1min":0.46800000000000003,"5min":0.192,"15min":0.192},"max":{"1min":0.76400000000000001,"5min":12.167,"15min":12.167},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Sep 13 07:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.51800000000000002,"5min":0.216,"15min":0.193},"max":{"1min":0.64800000000000002,"5min":2.1949999999999998,"15min":2.327},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.76600000000000001,"15min":0.67200000000000004},"min":{"1min":0.54700000000000004,"5min":0.26600000000000001,"15min":0.21099999999999999},"max":{"1min":0.73899999999999999,"5min":9.9469999999999992,"15min":9.9469999999999992},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197918720,"data_stored":3166699520,"data_compressed":14061120,"data_compressed_allocated":197918720,"data_compressed_original":3166699520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204738560,"data_stored":3275751443,"data_compressed":14545344,"data_compressed_allocated":204734464,"data_compressed_original":3275751424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402657280,"data_stored":6442450963,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:46:45.561 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:45.562 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-13T07:25:55.596315+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:45.562 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-13T07:25:55.596315+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:45.562 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T07:25:46.862483+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=25, tm_sec=46, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=7, tm_min=46, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T07:46:45.562 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T07:47:05.563 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T07:47:05.878 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T07:47:05.881 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T07:47:05.891 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-13T07:47:04.518102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443041358,"num_objects":1547,"num_object_clones":0,"num_object_copies":3094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":125716,"num_read_kb":4692374,"num_write":63654,"num_write_kb":15298218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66453,"ondisk_log_size":66453,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2789908,"kb_used_data":793468,"kb_used_omap":4,"kb_used_meta":1996411,"kb_avail":280325612,"statfs":{"total":289910292480,"available":287053426688,"internally_reserved":0,"allocated":812511232,"data_stored":12889265452,"data_compressed":57224892,"data_compressed_allocated":805380096,"data_compressed_original":12886048768,"omap_allocated":4782,"internal_metadata":2044325202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"2.7","version":"240'4260","reported_seq":10284,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:42.255162+0000","last_change":"2024-09-13T07:46:42.255430+0000","last_active":"2024-09-13T07:46:42.255162+0000","last_peered":"2024-09-13T07:46:42.255162+0000","last_clean":"2024-09-13T07:46:42.255162+0000","last_became_active":"2024-09-13T07:25:47.887500+0000","last_became_peered":"2024-09-13T07:25:47.887500+0000","last_unstale":"2024-09-13T07:46:42.255162+0000","last_undegraded":"2024-09-13T07:46:42.255162+0000","last_fullsized":"2024-09-13T07:46:42.255162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'4260","last_scrub_stamp":"2024-09-13T07:46:42.255106+0000","last_deep_scrub":"240'4260","last_deep_scrub_stamp":"2024-09-13T07:46:42.255106+0000","last_clean_scrub_stamp":"2024-09-13T07:46:42.255106+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:03:58.846624+0000","scrub_duration":0.0069620580000000001,"objects_trimmed":0,"snaptrim_duration":0.00097430100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15489,"num_read_kb":13863,"num_write":4046,"num_write_kb":6947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"240'3885","reported_seq":7884,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:41.218763+0000","last_change":"2024-09-13T07:46:41.219031+0000","last_active":"2024-09-13T07:46:41.218763+0000","last_peered":"2024-09-13T07:46:41.218763+0000","last_clean":"2024-09-13T07:46:41.218763+0000","last_became_active":"2024-09-13T07:25:47.887924+0000","last_became_peered":"2024-09-13T07:25:47.887924+0000","last_unstale":"2024-09-13T07:46:41.218763+0000","last_undegraded":"2024-09-13T07:46:41.218763+0000","last_fullsized":"2024-09-13T07:46:41.218763+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'3885","last_scrub_stamp":"2024-09-13T07:46:41.218712+0000","last_deep_scrub":"240'3885","last_deep_scrub_stamp":"2024-09-13T07:46:41.218712+0000","last_clean_scrub_stamp":"2024-09-13T07:46:41.218712+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:50.774341+0000","scrub_duration":0.0065878200000000003,"objects_trimmed":0,"snaptrim_duration":0.001148306,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9132,"num_read_kb":27041,"num_write":2612,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"240'8013","reported_seq":12434,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:46.322765+0000","last_change":"2024-09-13T07:46:46.323036+0000","last_active":"2024-09-13T07:46:46.322765+0000","last_peered":"2024-09-13T07:46:46.322765+0000","last_clean":"2024-09-13T07:46:46.322765+0000","last_became_active":"2024-09-13T07:25:47.889241+0000","last_became_peered":"2024-09-13T07:25:47.889241+0000","last_unstale":"2024-09-13T07:46:46.322765+0000","last_undegraded":"2024-09-13T07:46:46.322765+0000","last_fullsized":"2024-09-13T07:46:46.322765+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8013","last_scrub_stamp":"2024-09-13T07:46:46.322713+0000","last_deep_scrub":"240'8013","last_deep_scrub_stamp":"2024-09-13T07:46:46.322713+0000","last_clean_scrub_stamp":"2024-09-13T07:46:46.322713+0000","objects_scrubbed":0,"log_size":8013,"log_dups_size":0,"ondisk_log_size":8013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:15:26.052027+0000","scrub_duration":0.0072927549999999997,"objects_trimmed":0,"snaptrim_duration":0.00090095300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30453,"num_read_kb":24411,"num_write":10477,"num_write_kb":20366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"249'6030","reported_seq":11458,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:27.112265+0000","last_change":"2024-09-13T07:46:27.112265+0000","last_active":"2024-09-13T07:46:27.112265+0000","last_peered":"2024-09-13T07:46:27.112265+0000","last_clean":"2024-09-13T07:46:27.112265+0000","last_became_active":"2024-09-13T07:25:47.887818+0000","last_became_peered":"2024-09-13T07:25:47.887818+0000","last_unstale":"2024-09-13T07:46:27.112265+0000","last_undegraded":"2024-09-13T07:46:27.112265+0000","last_fullsized":"2024-09-13T07:46:27.112265+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6030","last_scrub_stamp":"2024-09-13T07:46:27.112008+0000","last_deep_scrub":"249'6030","last_deep_scrub_stamp":"2024-09-13T07:46:27.112008+0000","last_clean_scrub_stamp":"2024-09-13T07:46:27.112008+0000","objects_scrubbed":2,"log_size":6030,"log_dups_size":0,"ondisk_log_size":6030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:41.319842+0000","scrub_duration":0.015654925,"objects_trimmed":0,"snaptrim_duration":0.00055155999999999998,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12268,"num_read_kb":10256,"num_write":4577,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"249'6656","reported_seq":7779,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:32.991095+0000","last_change":"2024-09-13T07:46:32.991208+0000","last_active":"2024-09-13T07:46:32.991095+0000","last_peered":"2024-09-13T07:46:32.991095+0000","last_clean":"2024-09-13T07:46:32.991095+0000","last_became_active":"2024-09-13T07:25:56.610999+0000","last_became_peered":"2024-09-13T07:25:56.610999+0000","last_unstale":"2024-09-13T07:46:32.991095+0000","last_undegraded":"2024-09-13T07:46:32.991095+0000","last_fullsized":"2024-09-13T07:46:32.991095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6656","last_scrub_stamp":"2024-09-13T07:46:32.991074+0000","last_deep_scrub":"249'6656","last_deep_scrub_stamp":"2024-09-13T07:46:32.991074+0000","last_clean_scrub_stamp":"2024-09-13T07:46:32.991074+0000","objects_scrubbed":404,"log_size":6656,"log_dups_size":0,"ondisk_log_size":6656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T13:03:40.531248+0000","scrub_duration":2.8505866659999999,"objects_trimmed":0,"snaptrim_duration":0.00072277400000000003,"stat_sum":{"num_bytes":1690304512,"num_objects":404,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":857,"num_read_kb":1230055,"num_write":6463,"num_write_kb":3872623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"249'4232","reported_seq":11370,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:24.142391+0000","last_change":"2024-09-13T07:46:24.142391+0000","last_active":"2024-09-13T07:46:24.142391+0000","last_peered":"2024-09-13T07:46:24.142391+0000","last_clean":"2024-09-13T07:46:24.142391+0000","last_became_active":"2024-09-13T07:25:47.886191+0000","last_became_peered":"2024-09-13T07:25:47.886191+0000","last_unstale":"2024-09-13T07:46:24.142391+0000","last_undegraded":"2024-09-13T07:46:24.142391+0000","last_fullsized":"2024-09-13T07:46:24.142391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'4232","last_scrub_stamp":"2024-09-13T07:46:24.141846+0000","last_deep_scrub":"249'4232","last_deep_scrub_stamp":"2024-09-13T07:46:24.141846+0000","last_clean_scrub_stamp":"2024-09-13T07:46:24.141846+0000","objects_scrubbed":2,"log_size":4232,"log_dups_size":0,"ondisk_log_size":4232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T11:55:34.383253+0000","scrub_duration":0.017522368,"objects_trimmed":0,"snaptrim_duration":0.00059445099999999996,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15351,"num_read_kb":81517,"num_write":4129,"num_write_kb":39334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"249'6332","reported_seq":7379,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:55.020251+0000","last_change":"2024-09-13T07:46:55.020430+0000","last_active":"2024-09-13T07:46:55.020251+0000","last_peered":"2024-09-13T07:46:55.020251+0000","last_clean":"2024-09-13T07:46:55.020251+0000","last_became_active":"2024-09-13T07:25:56.611661+0000","last_became_peered":"2024-09-13T07:25:56.611661+0000","last_unstale":"2024-09-13T07:46:55.020251+0000","last_undegraded":"2024-09-13T07:46:55.020251+0000","last_fullsized":"2024-09-13T07:46:55.020251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6332","last_scrub_stamp":"2024-09-13T07:46:55.020205+0000","last_deep_scrub":"249'6332","last_deep_scrub_stamp":"2024-09-13T07:46:55.020205+0000","last_clean_scrub_stamp":"2024-09-13T07:46:55.020205+0000","objects_scrubbed":371,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:34.473826+0000","scrub_duration":2.6831274010000001,"objects_trimmed":0,"snaptrim_duration":0.0010219020000000001,"stat_sum":{"num_bytes":1556086784,"num_objects":371,"num_object_clones":0,"num_object_copies":742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":770,"num_read_kb":1012264,"num_write":6150,"num_write_kb":3673038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"249'4474","reported_seq":9376,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:39.276227+0000","last_change":"2024-09-13T07:46:32.232424+0000","last_active":"2024-09-13T07:46:39.276227+0000","last_peered":"2024-09-13T07:46:39.276227+0000","last_clean":"2024-09-13T07:46:39.276227+0000","last_became_active":"2024-09-13T07:25:47.886308+0000","last_became_peered":"2024-09-13T07:25:47.886308+0000","last_unstale":"2024-09-13T07:46:39.276227+0000","last_undegraded":"2024-09-13T07:46:39.276227+0000","last_fullsized":"2024-09-13T07:46:39.276227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'4474","last_scrub_stamp":"2024-09-13T07:46:32.232307+0000","last_deep_scrub":"249'4474","last_deep_scrub_stamp":"2024-09-13T07:46:32.232307+0000","last_clean_scrub_stamp":"2024-09-13T07:46:32.232307+0000","objects_scrubbed":2,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:14:33.145875+0000","scrub_duration":0.012268091,"objects_trimmed":0,"snaptrim_duration":0.000485625,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12406,"num_read_kb":27005,"num_write":4063,"num_write_kb":19318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"249'6228","reported_seq":7313,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:27.525411+0000","last_change":"2024-09-13T07:46:27.525591+0000","last_active":"2024-09-13T07:46:27.525411+0000","last_peered":"2024-09-13T07:46:27.525411+0000","last_clean":"2024-09-13T07:46:27.525411+0000","last_became_active":"2024-09-13T07:25:56.611595+0000","last_became_peered":"2024-09-13T07:25:56.611595+0000","last_unstale":"2024-09-13T07:46:27.525411+0000","last_undegraded":"2024-09-13T07:46:27.525411+0000","last_fullsized":"2024-09-13T07:46:27.525411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6228","last_scrub_stamp":"2024-09-13T07:46:27.525376+0000","last_deep_scrub":"249'6228","last_deep_scrub_stamp":"2024-09-13T07:46:27.525376+0000","last_clean_scrub_stamp":"2024-09-13T07:46:27.525376+0000","objects_scrubbed":352,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T12:07:04.530529+0000","scrub_duration":2.4237470619999999,"objects_trimmed":0,"snaptrim_duration":0.000575324,"stat_sum":{"num_bytes":1476395008,"num_objects":352,"num_object_clones":0,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":792,"num_read_kb":1043734,"num_write":6087,"num_write_kb":3719935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"240'3651","reported_seq":7604,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:29.185372+0000","last_change":"2024-09-13T07:46:29.185761+0000","last_active":"2024-09-13T07:46:29.185372+0000","last_peered":"2024-09-13T07:46:29.185372+0000","last_clean":"2024-09-13T07:46:29.185372+0000","last_became_active":"2024-09-13T07:25:47.887623+0000","last_became_peered":"2024-09-13T07:25:47.887623+0000","last_unstale":"2024-09-13T07:46:29.185372+0000","last_undegraded":"2024-09-13T07:46:29.185372+0000","last_fullsized":"2024-09-13T07:46:29.185372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'3651","last_scrub_stamp":"2024-09-13T07:46:29.185264+0000","last_deep_scrub":"240'3651","last_deep_scrub_stamp":"2024-09-13T07:46:29.185264+0000","last_clean_scrub_stamp":"2024-09-13T07:46:29.185264+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:20.455557+0000","scrub_duration":0.0074364490000000004,"objects_trimmed":0,"snaptrim_duration":0.00053109499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8893,"num_read_kb":24443,"num_write":2579,"num_write_kb":13833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"249'6819","reported_seq":8282,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:58.377356+0000","last_change":"2024-09-13T07:46:58.377637+0000","last_active":"2024-09-13T07:46:58.377356+0000","last_peered":"2024-09-13T07:46:58.377356+0000","last_clean":"2024-09-13T07:46:58.377356+0000","last_became_active":"2024-09-13T07:25:56.612381+0000","last_became_peered":"2024-09-13T07:25:56.612381+0000","last_unstale":"2024-09-13T07:46:58.377356+0000","last_undegraded":"2024-09-13T07:46:58.377356+0000","last_fullsized":"2024-09-13T07:46:58.377356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'6819","last_scrub_stamp":"2024-09-13T07:46:58.377303+0000","last_deep_scrub":"249'6819","last_deep_scrub_stamp":"2024-09-13T07:46:58.377303+0000","last_clean_scrub_stamp":"2024-09-13T07:46:58.377303+0000","objects_scrubbed":411,"log_size":6819,"log_dups_size":0,"ondisk_log_size":6819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:48:06.301062+0000","scrub_duration":3.0051236349999999,"objects_trimmed":0,"snaptrim_duration":0.00091467999999999996,"stat_sum":{"num_bytes":1719664659,"num_objects":411,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":1208,"num_read_kb":1165451,"num_write":6619,"num_write_kb":3892857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"249'5774","reported_seq":12099,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:51.309723+0000","last_change":"2024-09-13T07:46:51.310078+0000","last_active":"2024-09-13T07:46:51.309723+0000","last_peered":"2024-09-13T07:46:51.309723+0000","last_clean":"2024-09-13T07:46:51.309723+0000","last_became_active":"2024-09-13T07:25:47.890118+0000","last_became_peered":"2024-09-13T07:25:47.890118+0000","last_unstale":"2024-09-13T07:46:51.309723+0000","last_undegraded":"2024-09-13T07:46:51.309723+0000","last_fullsized":"2024-09-13T07:46:51.309723+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'5774","last_scrub_stamp":"2024-09-13T07:46:51.309697+0000","last_deep_scrub":"249'5774","last_deep_scrub_stamp":"2024-09-13T07:46:51.309697+0000","last_clean_scrub_stamp":"2024-09-13T07:46:51.309697+0000","objects_scrubbed":1,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T10:08:28.378981+0000","scrub_duration":0.013742420999999999,"objects_trimmed":0,"snaptrim_duration":0.00057946500000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17991,"num_read_kb":32243,"num_write":5715,"num_write_kb":15977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"249'99","reported_seq":394,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T07:46:26.071981+0000","last_change":"2024-09-13T07:46:26.072305+0000","last_active":"2024-09-13T07:46:26.071981+0000","last_peered":"2024-09-13T07:46:26.071981+0000","last_clean":"2024-09-13T07:46:26.071981+0000","last_became_active":"2024-09-13T07:25:42.741099+0000","last_became_peered":"2024-09-13T07:25:42.741099+0000","last_unstale":"2024-09-13T07:46:26.071981+0000","last_undegraded":"2024-09-13T07:46:26.071981+0000","last_fullsized":"2024-09-13T07:46:26.071981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'99","last_scrub_stamp":"2024-09-13T07:46:26.071916+0000","last_deep_scrub":"249'99","last_deep_scrub_stamp":"2024-09-13T07:46:26.071916+0000","last_clean_scrub_stamp":"2024-09-13T07:46:26.071916+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T08:13:31.823771+0000","scrub_duration":0.0066743669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":6442450963,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":3627,"num_read_kb":4451504,"num_write":25319,"num_write_kb":15158453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805314560,"data_stored":12884901926,"data_compressed":57212928,"data_compressed_allocated":805306368,"data_compressed_original":12884901888,"omap_allocated":0,"internal_metadata":0},"log_size":26035,"ondisk_log_size":26035,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":121983,"num_read_kb":240779,"num_write":38198,"num_write_kb":137871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40319,"ondisk_log_size":40319,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11964,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862532,"kb_used_data":195644,"kb_used_omap":1,"kb_used_meta":666878,"kb_avail":93509308,"statfs":{"total":96636764160,"available":95753531392,"internally_reserved":0,"allocated":200339456,"data_stored":3168350827,"data_compressed":14067102,"data_compressed_allocated":197955584,"data_compressed_original":3167272960,"omap_allocated":1594,"internal_metadata":682883526},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45700000000000002,"15min":0.47899999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.621,"5min":2.569,"15min":2.569},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.46300000000000002,"5min":0.183,"15min":0.183},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Sep 13 07:46:16 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.54000000000000004,"15min":0.56000000000000005},"min":{"1min":0.51200000000000001,"5min":0.222,"15min":0.216},"max":{"1min":1.1919999999999999,"5min":3.593,"15min":3.6389999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.52500000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.97999999999999998,"5min":5.718,"15min":5.718},"last":0.46899999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705925,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1116488,"kb_used_data":395520,"kb_used_omap":1,"kb_used_meta":720958,"kb_avail":93255352,"statfs":{"total":96636764160,"available":95493480448,"internally_reserved":0,"allocated":405012480,"data_stored":6443511875,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":1594,"internal_metadata":738261446},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.438,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.186,"15min":0.185},"max":{"1min":0.67300000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.51400000000000001,"15min":0.55900000000000005},"min":{"1min":0.47299999999999998,"5min":0.217,"15min":0.191},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":2.153},"last":0.59999999999999998}]},{"osd":1,"last update":"Fri Sep 13 07:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.29399999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68300000000000005,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.501},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.48199999999999998,"15min":0.55500000000000005},"min":{"1min":0.33100000000000002,"5min":0.23000000000000001,"15min":0.22500000000000001},"max":{"1min":0.85699999999999998,"5min":1.1819999999999999,"15min":2.4689999999999999},"last":0.54200000000000004}]}]},{"osd":1,"up_from":9,"seq":38654705923,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":810888,"kb_used_data":202304,"kb_used_omap":1,"kb_used_meta":608574,"kb_avail":93560952,"statfs":{"total":96636764160,"available":95806414848,"internally_reserved":0,"allocated":207159296,"data_stored":3277402750,"data_compressed":14551326,"data_compressed_allocated":204771328,"data_compressed_original":3276324864,"omap_allocated":1594,"internal_metadata":623180230},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 07:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.41599999999999998,"15min":0.47699999999999998},"min":{"1min":0.442,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.69899999999999995,"5min":0.92700000000000005,"15min":2.363},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.67200000000000004,"15min":0.58799999999999997},"min":{"1min":0.46800000000000003,"5min":0.192,"15min":0.192},"max":{"1min":0.76400000000000001,"5min":12.167,"15min":12.167},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Sep 13 07:46:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.51800000000000002,"5min":0.216,"15min":0.193},"max":{"1min":0.64800000000000002,"5min":2.1949999999999998,"15min":2.327},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.76600000000000001,"15min":0.67200000000000004},"min":{"1min":0.54700000000000004,"5min":0.26600000000000001,"15min":0.21099999999999999},"max":{"1min":0.73899999999999999,"5min":9.9469999999999992,"15min":9.9469999999999992},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5982,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":27,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197918720,"data_stored":3166699520,"data_compressed":14061120,"data_compressed_allocated":197918720,"data_compressed_original":3166699520,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204738560,"data_stored":3275751443,"data_compressed":14545344,"data_compressed_allocated":204734464,"data_compressed_original":3275751424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402657280,"data_stored":6442450963,"data_compressed":28606464,"data_compressed_allocated":402653184,"data_compressed_original":6442450944,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T07:47:05.893 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T07:47:06.226 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T07:47:06.226 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T07:47:06.226 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T07:47:06.226 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T07:47:07.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-13T07:47:12.328 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T07:47:12.329 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T07:47:12.329 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T07:47:12.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-09-13T07:47:17.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-13T07:47:17.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-13T07:47:18.431 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T07:47:18.431 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T07:47:18.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T07:47:23.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-13T07:47:23.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-13T07:47:24.532 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T07:47:24.533 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T07:47:24.533 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T07:47:24.533 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T07:47:24.569 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T07:47:24.569 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T07:47:24.570 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T07:47:24.570 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T07:47:24.620 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T07:47:24.620 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T07:47:24.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-13T07:47:24.638 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:47:24.638 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T07:47:24.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:47:24.884 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T07:47:25.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:47:25.085 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T07:47:25.335 INFO:tasks.ceph:Archiving mon data... 2024-09-13T07:47:25.336 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930/data/mon.a.tgz 2024-09-13T07:47:25.337 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T07:47:25.341 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.V5VPoTPk48 2024-09-13T07:47:25.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.V5VPoTPk48 2024-09-13T07:47:25.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.V5VPoTPk48 2024-09-13T07:47:25.471 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.V5VPoTPk48 is 445KB 2024-09-13T07:47:25.536 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.V5VPoTPk48 2024-09-13T07:47:25.543 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T07:47:25.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T07:47:25.559 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-13T07:47:25.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T07:47:25.633 INFO:teuthology.orchestra.run.smithi005.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T07:47:25.640 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T07:47:25.641 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930/remote/smithi005/crash 2024-09-13T07:47:25.642 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T07:47:25.659 INFO:tasks.ceph:Compressing logs... 2024-09-13T07:47:25.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T07:47:28.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-13T07:48:27.862 INFO:tasks.ceph:Archiving logs... 2024-09-13T07:48:27.862 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930/remote/smithi005/log 2024-09-13T07:48:27.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T07:48:37.121 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T07:48:37.149 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T07:48:37.149 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T07:48:37.191 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T07:48:37.200 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T07:48:37.200 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T07:48:37.221 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T07:48:37.221 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-09-13T07:48:37.221 INFO:teuthology.orchestra.run.smithi005.stdout:*hv01.front.sepi 67.205.162.81 3 u 62 64 377 0.092 -0.493 0.146 2024-09-13T07:48:37.221 INFO:teuthology.orchestra.run.smithi005.stdout:+hv02.front.sepi 96.245.170.99 3 u 9 64 377 0.100 2.317 0.231 2024-09-13T07:48:37.221 INFO:teuthology.orchestra.run.smithi005.stdout:+hv03.front.sepi 74.6.168.72 3 u 3 64 377 0.081 0.139 0.199 2024-09-13T07:48:37.221 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-13T07:48:37.222 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T07:48:37.231 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T07:48:37.231 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T07:48:37.243 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T07:48:37.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T07:48:37.265 INFO:teuthology.task.internal:Duration was 1852.609654 seconds 2024-09-13T07:48:37.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T07:48:37.303 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T07:48:37.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T07:48:37.355 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T07:48:37.356 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T07:48:37.356 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T07:48:37.369 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T07:48:37.369 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T07:48:37.399 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T07:48:37.400 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T07:48:37.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T07:48:37.476 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T07:48:37.477 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T07:48:37.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T07:48:37.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T07:48:37.548 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-09-13T07:48:37.564 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T07:48:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T07:48:37.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T07:48:37.580 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T07:48:37.580 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_22:16:02-rbd-reef-distro-default-smithi/7900930/remote/smithi005 2024-09-13T07:48:37.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T07:48:37.605 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T07:48:37.605 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T07:48:37.648 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T07:48:37.658 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T07:48:37.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T07:48:37.667 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T07:48:37.667 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T07:48:37.675 INFO:teuthology.orchestra.run.smithi005.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 07:48 /home/ubuntu/cephtest 2024-09-13T07:48:37.677 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T07:48:37.686 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T07:48:37.716 INFO:teuthology.run:Summary data: description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_20.04} workloads/rbd_cli_generic} duration: 1852.609653711319 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-13T07:48:37.716 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T07:48:37.811 INFO:teuthology.run:pass